{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression：linear model\n",
    "这里采用最简单的linear model：**y_data=b+w*x_data**\n",
    "\n",
    "我们要用gradient descent把b和w找出来\n",
    "\n",
    "当然这个问题有closed-form solution，这个b和w有更简单的方法可以找出来；那我们假装不知道这件事，我们练习用gradient descent把b和w找出来"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 数据准备："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 假设x_data和y_data都有10笔，分别代表宝可梦进化前后的cp值\n",
    "x_data=[338.,333.,328.,207.,226.,25.,179.,60.,208.,606.]\n",
    "y_data=[640.,633.,619.,393.,428.,27.,193.,66.,226.,1591.]\n",
    "# 这里采用最简单的linear model：y_data=b+w*x_data\n",
    "# 我们要用gradient descent把b和w找出来"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 计算梯度微分的函数getGrad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 计算梯度微分的函数getGrad()\n",
    "def getGrad(b,w):\n",
    "    # initial b_grad and w_grad\n",
    "    b_grad=0.0\n",
    "    w_grad=0.0\n",
    "    for i in range(10):\n",
    "        b_grad+=(-2.0)*(y_data[i]-(b+w*x_data[i]))\n",
    "        w_grad+=(-2.0*x_data[i])*(y_data[i]-(b+w*x_data[i]))\n",
    "    return (b_grad,w_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1、自己写的版本\n",
    "当两个微分值b_grad和w_grad都为0时，gradient descent停止，b和w的值就是我们要找的最终参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the function will be y_data=-inf+nan*x_data\n",
      "the average error is nan\n"
     ]
    }
   ],
   "source": [
    "# 这是我自己写的版本，事实证明结果很糟糕。。。\n",
    "# y_data=b+w*x_data\n",
    "# 首先，这里没有用到高次项，仅是一个简单的linear model，因此不需要regularization版本的loss function\n",
    "# 我们只需要随机初始化一个b和w，然后用b_grad和w_grad记录下每一次iteration的微分值；不断循环更新b和w直至两个微分值b_grad和w_grad都为0，此时gradient descent停止，b和w的值就是我们要找的最终参数\n",
    "\n",
    "b=-120 # initial b\n",
    "w=-4 # initial w\n",
    "lr=0.00001 # learning rate\n",
    "b_grad=0.0\n",
    "w_grad=0.0\n",
    "(b_grad,w_grad)=getGrad(b,w)\n",
    "\n",
    "while(abs(b_grad)>0.00001 or abs(w_grad)>0.00001):\n",
    "    #print(\"b: \"+str(b)+\"\\t\\t\\t w: \"+str(w)+\"\\n\"+\"b_grad: \"+str(b_grad)+\"\\t\\t\\t w_grad: \"+str(w_grad)+\"\\n\")\n",
    "    b-=lr*b_grad\n",
    "    w-=lr*w_grad\n",
    "    (b_grad,w_grad)=getGrad(b,w)\n",
    "\n",
    "print(\"the function will be y_data=\"+str(b)+\"+\"+str(w)+\"*x_data\")\n",
    "\n",
    "error=0.0\n",
    "for i in range(10):\n",
    "    error+=abs(y_data[i]-(b+w*x_data[i]))\n",
    "average_error=error/10\n",
    "print(\"the average error is \"+str(average_error))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的数据输出处于隐藏状态，点击即可显示"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2、这里使用李宏毅老师的demo尝试"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 引入需要的库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "matplotlib.use('Agg')\n",
    "%matplotlib inline \n",
    "import random as random\n",
    "import numpy as np\n",
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 准备好b、w、loss的图像数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 生成一组b和w的数据图，方便给gradient descent的过程做标记\n",
    "x = np.arange(-200,-100,1) # bias\n",
    "y = np.arange(-5,5,0.1) # weight\n",
    "Z = np.zeros((len(x),len(y))) # color\n",
    "X,Y = np.meshgrid(x,y)\n",
    "for i in range(len(x)):\n",
    "    for j in range(len(y)):\n",
    "        b = x[i]\n",
    "        w = y[j]\n",
    "        \n",
    "        # Z[j][i]存储的是loss\n",
    "        Z[j][i] = 0\n",
    "        for n in range(len(x_data)):\n",
    "            Z[j][i] = Z[j][i] + (y_data[n] - (b + w * x_data[n]))**2\n",
    "        Z[j][i] = Z[j][i]/len(x_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 规定迭代次数和learning rate，进行第一次尝试\n",
    "距离最优解还有一段距离"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAd1ElEQVR4nO3de5Bc5Znf8e8zo7s0FkKAMDB4LNaI61oSchaBWR8DXuxsbGphE7JxSsnCImezyebmcnAwBlNbtTaFqRAqYCuLk2Wr4g1s2disbWIHfEBlJAcJqIAxwsiMPBaLuOjWGmmYGfWbP6ZHMxpmevpyznnP+/bvU0Vp+mj69KNjub96+5zuMeccIiIiM+nyPYCIiJSbQiEiInUpFCIiUpdCISIidSkUIiJS1xzfAzTqpJNOcn19fb7HEBEJyvbt299yzp3czj6CCUVfXx/btm3zPYaISFDMbFe7+wgmFK+9NsJtt73ue4xcpOncXPefJItz3b+IlMttty3IdH/BhCJmSTKS6/7TdDDX/fug+IkUR6HoAHmHyAfFT6Q4CoUESfErP4UvHgqFSEnEFj+FLx4KhYjkQuHzSSezRUQKF1v4mqF3ZouISF3BrCgqlVHS9M2W758kbb0xUUSkYwUTip4eI0m6W75/O5EpK8VPRIoQTCja1U5kykrxE5EidEwoYqT4lZ/CJzFQKKRU4ovfXtL0qO8hMqPwdSaFQiRnccVP4etECoWINCWm8MX0Umee0VMoRKRjxRu9UzPdt0IhIhKBPKPn/Z3ZZrbCzJ71PYeIiEzPeyiAO4GFvocQEZHpeQ2FmV0ODAJx/oxTEZEIeDtHYWbzgFuA3wMenuF7NgIbAebPP4U0Hai7zyTpzXhKERHxeTL7JuBe59x+M5v2G5xzm4BNAKeddo5Lkp5ZdrmfNK1kO2VJKIIi4ovPUFwJXG5mfwKsNrO/cM79Ubs7nT0mYZptNRUahU8kHN5C4Zz77fGvzSzNIhIxiy2ACp9IOErxPgrnXOJ7BilWbOGL7WVPhU8mK0UoRGIQU/xiWvEpeu1TKETkXRS98vEZPIVCRKIWS/SaC955mT62QiEiEgCfwSvDR3iIiEiJKRQiIlKXQiEiInUpFCIiUpdCISIidSkUIiJSl0IhIiJ1KRQiIlKXQiEiInUpFCIiUpdCISIidSkUIiJSl0IhIiJ1KRRNWrV4M0a17vcYVVYt3lzQRCIi+VIompAsv58/OP0mPrXiyzPGwqjyqRVf5g9Ov4lk+f0FTygikj2FokGrFm8mWf4NANYs/d60sRiPxJql3wMgWf4NrSxEJHgKRYNeHryUZw/87rHbU2MxNRIAzx74XV4evLTwWUVEsqSfcNcgRxff3XMTwLEYjP/6yJ7P8ckVd7wrEt/dcxNOLRaRwCkUTZgpFpMDAYqEiMRFz2RNGo/F5JehJlMkRCQ2ejZrgaOLR/Z8btrfe2TP5xQJEYmKntFaYFT55Io7pv29T664Y9b3WYiIhETnKJo03dVNk41v18tPItn40pdav3Lw1lt/kuEknUvPZE2Y6RLY219+ou6lsyLSmnYikcX9ZYxWFA2aKRLjK4eZLp3VykJEQqdnsAadvfgndd8nMd3VUGuWfo+zF2vpKyJhUygatGPwMtK3rwdmvgR2aizSt69nx+Blhc8qEgudYygHvfTUhPTtG/i7obN5efDSGV9OGo/FS4cuUyREMtBMLO6885MMDu49dnvx4hPzGKnjaEXRpB2Dl816zsHRpUiIeLB8+Zl1b0trFAoRicbChe+pe1tao1CIiEhdCoWIiNTl9WS2mS0F/hroBgaB65xzwz5nEpFwLVmyrO5taY3vFcWngbucc78DvA583PM8IhKw9753Vd3b0hqvKwrn3L2Tbp4MvOFrFhEJ3+HDBybdsim3pVW+VxQAmNl6YJlzbuuU7RvNbJuZbTt8eL+n6UQkFIsWLZ10y025La3yHgozOxG4B7h+6u855zY559Y559YtWnRC8cOJSFC0osiH11CY2TzgIeDzzrldPmcRkfBpRZEP3yuKG4C1wM1mlprZdZ7nEZGAaUWRD98ns+8D7vM5g4jEo69vLWCAo7t7Tu22tMv3ikJEJGPO9wDRUShEJBr9/c8c+7paPXrcbWmdQiEi0ejrW4uZAdDdPVcvPWVEP49CRKLR23sBy5adzuHDB7jyyj+mt/cC3yNFQSsKEYnGwMAL7Nu3m6GhCo8+ejcDAy/4HikKCoWIRKO//xmcGzuZffToiM5RZCSYl54qlSppWmnoe5OkJ+dpRKSMxs9ROOd0jiJDwYSip2ceSdLb0Pem6UDO0xRP8ROZnc5R5COYUDSj0aCEJMb4jVMEJSvj5yicczz66N2ccspZikUGogxFjGKM37hYI6gAFm+6cxQKRfsUCvEu1gjGGsDJyhZDnaPIh0IhkpNYAzhZ+WL4PhYtej+jo4Oce+5/ZOfO97FzZ2MXwUxVtgj6pFCISMvKGMP+/uXAcq6++qq291W+EPqhUIhIVN555xBDQwcZGHiO3t7Vbe2rjCH0QW+4E5FoDAw8x549L7F//24eeOAGBgae8z1SFBQKEYlGf//TU656etrzRHFQKEQkGn19H5ry6bEf8jxRHHSOQkSi0du7mhUrzmFo6CDXXHNH2+coZEwwoahUHGl69LhtSdLtaRoRKav585cwf/4SRSJDwYSip2cOSXLycdvS9E1P0+RLARRpXZZXPcmYYEIxnanhiEWMAVT8pAjjVz0553jggRvYsOF+xSIDQYciVgpgWBTB8pjuqieFon0KhRQm1gCCIlgW41c9TXzWk656yoJCIZKBWCMYXgAvZNGicxgdPcC5536FnTsvZOfOozN+d2gh9EWhEJEZhRjAXbuW4dxSrr76ylm/N7wQ+qFQiEjHCjGEPuid2SIiUpdCISJRGRqqcODAbgYGtvkeJRrBvPRUqRhpOtf3GC1LkhHfI4hEb2BgG3v2/AznqjzwwD9kw4aH6O1d53us4AUTip6ebpJkse8x2pKmg75HyJwCKGXS3/8UzlWB8fdRPKVQZCCYUMQg9NDNJMYAgiIYor6+SzDrwrlq7X0Ul/geKQoKhbRNAQxLzAHs7V3HihXnc+TIPq699j6tJjLSUCjMzNz4++JFOkSsAYR4IwiwYMF7mDdvkSKRoUZXFK+Y2d3AN5xzh/IcSETyF3ME777bcC7si19m4ms12Ggovg3cCnzJzP4bcI9zbiC/sUREWnPCCd1Uqy7KGPpaCTYUCufcZ83sFuCfAX8C/Fsz+xbwVedcIT+UtlKBNJ193CQZLWAaEZHi+YpfwyeznXNHgK8BXzOzK4A/BZ4ys63AXc65b7cygJndD5wHfM8592czfV9Pj5Eks4+bpq1MUX4KoEhjhoYOcOTIPgYGttLbe7HvcaLQ6Mns04BlU/57HDgI/GPgb4CmP4bRzK4Bup1z683sG2b2AefcL5rdz2SNxCREMQZQ8ZOsDQxsZc+e/1d7w93fZ8OG7ysWGWj0WfXXQBU4BFSAA4xF4gDwrdqvrUiAB2tf/xD4MHAsFGa2EdgIsHTpmS0+RBziDOAc0jTOWCiCfvT3b570hrth+vs3KxQZaPTZ51ngbOB/AP/ZOdef0eMvBnbXvt4LrJ38m865TcAmgNNOW6fLcyMUZwBBEfSjr++ySW+4m0df32W+R4pCoyezLzKzjwL/AdhhZg8zdiL7/7b5+IeAhbWvl6APKZSIxBrBcr8M+mEWL/4gIyN7Offcv2LnzvXs3Nn4vcscQZ+aOZn9Y+DHZnYuY8F4wsy2A3cC32nxDXnbGXu5aSvwQWBHC/sQkQKVPYC7dp1AtbqEq69ufjVR7gj60/T/4s65n5vZjcAdwH9i7BzDLuADLTz+w8Dm2snyTwAzvphYOQzpcy08wiwS/dx1EakpewR9afSqp+2MvTTUA7yHiZeLrPbrylYe3Dl30MwS4GPAHc65GU+K9/RAkrTyKPXF+i8IBVA6m05pZqnRfD4D7Jvlv5Y45/YxceVT4fKITxkogNKpzGz2b5KmNHoy+8a8B5FsKYBhUQClzPSCnAQl1gBCnBFUAOOgUIiURKwRLDqA+ypQreZz8ctUnRLCYEJRGYL0xdbum5yX7Swi0riiA/jy9gMcHnyLs/q20Nu3PtfHinEVOJ1gQtGzGJI23omfbs1ulrJQAEWON9C/hdd3Pzv2WU/3XsGGf/lYrrGIdRU4VTChaFc7kSkzBVBkQv8r6fGf9fRKmvuqohN0TChiFWMAY4zfOEUwX32/kRz/WU+/kfgeKQoKhZROjPEbF2sEyxLA3r71nHr6Gg4PvsW1G76p1URGFAqRAsUawTIF8Eh1KW7eQnYeXs/OFi+AmawsEfQpmFBU3oG0iU+BzFtylu8JRMqjTAH81WY4ejS7mcoUQV+CCUXPIkjW+J5iQvqs7wnyoQBKDFr7MOvplSmCvgQTirIpU7SypABK8PRZT5lTKOQ4CmBYFEApgkIhHUEBDIsCWC7BhKIyAulr+ew7OS2f/YrkLdYAQusR3HcEqqPluvhlshAjGEwoehbke5laq58jVWYKoISs1Qj+6hEYHS5vRENcBQYTirzFeq20AiidZmjwAIcPvsnAji30rirfG+7KGrB6FIrIxRjAGOMHCmAWBnZs4fVXnxn7UMDbr2DDFx8rZSxCo1BIcGKM37gYI1hkAPtfnPShgKPD9L+YKhQZCCYUlVFI3yr2MZOTin08kVgjWFQAD5ySgHWBq2Ld89h7SpLbRTDQOavAYELRMx+SlcU+ZvrLYh+vCIqf+FBYAM9bz57vr+Xw/je59uZv0nte/quJGFeBUwUTCh+KDlMRYowfKIAyYcHipcyZM6+QSEC8q8DJFIoOE2P8QAEUyZNCIVFQAMOSZwBNn/WUuWBCUalCWvE9ReuSHt8TSIhiDSDkF8G9w7V3Zhd88cu4GFeBwYSiZy4kp/ueonXpbt8TZE/xk3bkFcGBRTDS5Teysa0EgwlF6EKOXD0KoMi7xbYSVCikLQpgOBRAaVUwoag4SEfzf5wkmCMieVIAw6II5iuYp8WeOcWcJPJ1AixPip+M64QA7h2F6tGwL36ZrAwR1FPIFDFesRBj/EABlAmTAzgwH0ZcPFEswypQ/1frADHGDxRA6QxlCF4wfyUPmWNz91DT97vs6IIcppEyUADDowiGKZj/2Xq6IFnc/P3SwebjEgIFMF6xBhCKieAbB/cz8vYbfGfbFpauLujznoJ5Jm2Nlz+emS0F/hroBgaB65xzw3k8VitxCUGMAVT84pd3BAe2beHJnz+Dq1Z54cYr2PDQY/SuK+ATZCNeBYK/FcWngbuccz8ys/uAjwPf9TRLkOIM4BDpoO8ZsqcAFqf/qRRXrf3gopFh+p9KCwlFzKtA8BQK59y9k26eDLwx230GbZRtcw80/BjrRpa2MJn4pgCGpWwR7Lskwbq6cNUq3XPn0XdJ4nukKBQSCjP7OrBq0qbHnXO3m9l6YJlzbusM99sIbARYduYZfGRu4+M+QeNRCYXiF644A1jCl0B/aw1LfnMtw2++zgWb/or+D62hH10E065CQuGc+8zUbWZ2InAPcG2d+20CNgH0rlvtmnnMZqISjLmDPDFSwNvTC6YAhquMAdx14lKG53dzdXJxG3sJfRWYbeh8ncyeBzwEfN45t8vHDKGKMYAxrv5AAQxdGSPoi69nnRuAtcDNZnYzcJ9z7n/Vu8MRRnlh9lMZs7qAU9reh2QrxvgBWgFKNHydzL4PuK+Z+yyhi/UsbPuxt2QQm7JR/MorxgiWfQV4sGuUUTva1MUv4xTB6cX3t3gWWcSmfCps4YjvITKnAJZT2eP3khnvmLU0Z9kj2LgIzlFI9mIMYIyrP1AAy6zsEfQlmKMyxDA78PMxiqsowadydaAY4wcKoIQnmFAsopu1nODlsZ/xFKi8KYB+KID5OsQwI244k4tfxnV6BIMJhU++ApU3BVCyVJYA/tS6Gcro4pdxZYlg45ZnujeFooMpgGFRAP0pSwR9CSYU7zDML3nV9xiZWMn7fY8QNQUwLApg+QUTioXM4cJIXid8PpLgTaUA5ksBbMwg7zDs8eKXcTEFMJhQxCSW4E2lAEorsg7gD5nDEeZ4D6vfFWBfpnsLJhQjDPEaL+Wy79M4J5f9dhoFMCwKYL58hypLwYRiAfNyXMpVvC9T86AAZkMBlE4XTCjyFtPrieN25LQC800BzEasAfza/rc49Hd7Sbf8gDPXx/N3xecKMJhQjHLY9wjBiTF+oADKzF7a8jyvbXsFV63yl1d8gdsf+6+cs/5C32NlZJDnG34/x/mZPnIwoZjHfJZwiFcjXC6fTCx/kYuhAIalyAC+kG4/9jOzR4dHeSHdHlEo/K0CgwnFuPdHeALuVZ73PUIuFMDmKIDtOzU5CesyXNXRPa+bU5OTdBFMBoILRYxijB/EGUDFr3lFBnDV+tP523UP8vZre7nlwVs4f/15OT5amS+C+XuZ7i2YUBzlMPt5xvcY73ICa32PUFpxBjDOlz8hngj2LOvBzHKOxJhYV4FTBROKuSzgdFb5HuNddpcwXllQAGcWZwDjWQEOc4gRjvBm7c8TSwB9CiYUZVXGeGUhxgAqfvXFEsCFLGSU6rE/TywBnKrIAAYTiiqDHGZrro+xiItz3X9I4gzgILvZ4XuIzCmA9cUSwHer9zLoFZk+UjChmMvCAgo6eGy5GhMFcIICGA4FcHZFRTCYUBQlxtcz38x5JeaLAjhBAZwwwkGOMlTKi1/GhRbBYELhqHCUtK19dJNkMktoYowfKICdoJUALmAxIy3etyj5nwO8OtO9BROKbhZxAhe1tY/9bYamrBTAuCiA8StzxKYTTCiy0G5oyirGAHZq/EABbNdR9lPlndwvfhnXCQEMJhTGQbqq/6ep+1S7rsxpmnKJM4AV9rPd9xC56NQIFhXAeTzOHLoLe7xyrgD/SaZ7CyYUXSxhIeubus+RJsMSgk6JH8QaQK0AYxPrCnCyYELRimbDEoTqIEfY4nuKzCmAoSvPCtCxF+eG2774ZVwnR3BcOKE4epA5Bx9v6i6j77k8p2H8UgDDoQAWb549QzeHM5snzBXgjZnuLZhQGD3MsY80d6cmwxKCWOMHcQYwxpc/QQHsNMGEohVNhyUEEcYP4g1gjPEDyr0CdG8BI01f/DKukyLYqHBCMVKBPWnr91+RZDWJV1HGDxTAAJU1gt38gi6OtDxfFKvArn+f6e7CCUVXDyxIWr9/O5Epq0jiBwpgaBTAzhJOKNrVTmTKKsb4gQIYghIH0Eb3YkdHmr74BeIOYDvCCcVwBX6dZr/fM5Ls91mUGOMHCmAAyhxAs19jHGltxhIHsCkn3JLp7ryGwsxWAI8659bM+s3dPbA0yX6IPOLjW8jxAwUwNApg9HyvKO4EFnqdII/4+FYBDqS+p8ieAlhOJQvgvjf38Nqed9jy6HdYv2Zp6zuKKIDt8hYKM7scGAReb+gOwxX4ZZrnSDNbmfh53HYogOFQADOzZfsATz//JNWq44p//gKP/c8NrL+ot7WdlSyATWmjj9MpJBRm9nU47nN1Hwc+Cvwe8HCd+20ENgKceepSODnJcco6fAUqbysT3xM0L8YAxvjyJ3gJYLq1H1d1AAyPHCXd2t96KEoUQN8KCYVz7jOTb5vZF4F7nXP7zaze/TYBmwDWnXuay3XIenwFKm8KYDnEGD/wEsDkrAMsmN/F8EiVeXOM5Ky92c8R+gqwBeZc8c+/ZvYkUK3dXA38jXPuj+rdZ11fj9t2S0BvpT8/8T1B53oz9T1BPlYmvicIwpbnBkif7if5UB/rV7e4mphN2V8CvSQ99qWZbXfOrWtnd15CcdwAZqlzLpnt+9adfZrbds/GAibKSIwvJyh+fimA0qjzbzv2ZRah8H3VE41EAoDDFXguPX7b6sbu6kWMy9Ofpb4nyEcoAdRLoGFZmfieIDPeQ9Gw+T1wVnL8tqnhiIHiVzwF0C8FMHvnZ7u7cEIxnanhiEEF2Jn6niJ7CmDxFEC/IgpgOKEYrMBP0+z3+1tJ9vtsV4wBjHH1BwqgDwpg4cIJxYIeOC/Jfr8V4MU0+/36VrYAxhg/UAB9iDWAB8juIpiLs9nNuHBCkac8AuRbHquvMlAAi6EA+lHSCIYTikMVeCL1PcXsPpL4nmBMjPEDBbAoCmBYcg5gOKFY1ANrEt9TzO4g8Gzqe4rsKYD5UgCL0SkBvCrb3QcTCnewwtCP0oa+d8HHklxnmVUIQWtWCKu5ViiA+VIAi5FzAIMJBYt74OKkoW9tNCih8RrAGOMHCmDeFEA/fj/b3YUTimY0GJTQKIA5UADDogB6EUwoRioV3no89T3GMSddnhT/oApgMPTyZ04UQC+CCYUt6WHObye+xzimTNHKipf4QZQBHKoAW1PfY2ROAcxJ1gH8w2x3F0wohisVfpWmdb/nzCQpZBagVNHKyv5RGH0y9T1G5hTA7CiAOSl5AIMJRXdPD0tmCcFsIQlRkfGDOAMY4+oPFMAsxRbABf8m2/0FE4pGzBaSEO0FDimAbYkxfqAAZi7CAGYlmFAMVSr8ooRPmB8o4AkvxgDGuPoDBTALCmD5BBOKOT09nFzCJ8wyxms6RQStGTHGD8IKYNEvazYq1gAWeQ7wjC9kuz/vPwq1UWZWAXb4nqMkTgLe8j1ESehYTNCxmKBjMWGVc66nnR0Es6IAdrT7c19jYWbbdCzG6FhM0LGYoGMxwcy2tbuPriwGERGReCkUIiJSV0ih2OR7gBLRsZigYzFBx2KCjsWEto9FMCezRUTEj5BWFCIi4oFCISIidZUuFGa21Mx+YGY/NLNvm9m82vb7zWyLmX1h0ve+a1tszGyFmW2edHulmT1mZs+Z2S21bXPN7BEz+4mZXe9v2nxNPRaTtj9iZqtrX3f6sbjAzH5U+7ojj4WZnWlmqZk9bmabbEynHot3/blbORalCwXwaeAu59zvAK8DHzeza4Bu59x6YKWZfWC6bR5nzoWZLQP+Elg8afO/Ar7onFsNXGVmJwP/GtjunLsU+H0za+vNNWU0w7HAzD4N7HTOPVfb1MnHwoC7gLm1TZ16LD4D/LFz7nKgF7iQzj0W0/25mz4WpQuFc+5e59yPajdPBt4AEuDB2rYfAh+eYVtsjgLXAQcnbXsb+E0zWwHMB/Zz/LF4EojxjUbvOhZmdiLwVWCfmX20tjmhA49FzR8CP550O6EDj4Vz7mbn3M9rN5cz9g7thA48Fkz/555uW13e35ltZl8HVk3a9Lhz7nYzWw8sc85tNbMbgd21398LrGWsmlO3Ba3OsZj8bY8CfwqcATwOjPLuY7Ei/2nz1eCx+HfAQ8DXgT+v/cuoI4+FmS0H/ilwVe0/6NBjMel7rwN+5px7zcw69VhM9+du+lh4D4Vz7jNTt9X+pXgPcG1t0yFgYe3rJYythKbbFrTpjsU0bgL+kXPOmdl/AT7GxLE4wNixOJTflMVo8FisAT7rnHvdzB6ks4/Fl4HPO+dGJj1RdOqxwMxWAp8Frqxt6tRjMd2fu+ljUbon19rJ64cY+0u/q7Z5OxMvLX0Q6J9hWyd4P9BrZgsYW0U5OvdYvAKsrH29DthF5x6LjwBfMbMUWG1mf0aHHovaa/XfBK53zh2obe7IY0FGz53eVxTTuIGxJ8Cbzexm4D7gYWCzmZ0GfAK4mLEnyKnbOsGtQMrY+Zu/Zezlp5eB75vZZcB5wE+9TVesO4C/qP09OQxcA5xIBx4L59zZ41+bWeqc+4KZvY8OPBaMrbrPBO6pra5uZewkbycei+n+3Lun2VZXMO/Mrv0r4WPAk86512fa1qlqwfww8L8n/SuqI+lYTNCxmNCpx2K6P3ezxyKYUIiIiB+lO0chIiLlolCIiEhdCoWIiNSlUIiISF0KhYiI1KVQiLTIzM42M2dmn/A9i0ieFAqR1o1/vtjTXqcQyZlCIdK6i4B+59xbvgcRyZNCIdK6i4CnzWyjmb1oZoO1H6QV/CcZi0ymd2aLtMjM9jH2GVPPMPaZZIuBrzD2c0LOds4NehxPJDNl/FBAkdIzs7OAExj7QLVPudq/uMysAvwAuJSxH6glEjy99CTSmotqv97sjl+WP1/79eSC5xHJjUIh0pq1jP2s7u1Ttr+39uuvC55HJDcKhUhrLgJem2b7dYz95LAtxY4jkh+doxBpzVpgyMzmOOdGAczsDOBfAF91zg17nU4kQ7rqSaRJZtYHvFr77yngvwNnALcx9tPDLlcoJCZ66UmkeeMnsv8BYz+k/hHgy8C3gKsUCYmNVhQiIlKXVhQiIlKXQiEiInUpFCIiUpdCISIidSkUIiJSl0IhIiJ1KRQiIlLX/wfnvKHJdJERAAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# y_data = b + w * x_data\n",
    "b = -120 # initial b\n",
    "w = -4 # initial w\n",
    "lr = 0.0000001 # learning rate\n",
    "iteration = 100000 # 这里直接规定了迭代次数，而不是一直运行到b_grad和w_grad都为0(事实证明这样做不太可行)\n",
    "\n",
    "# store initial values for plotting，我们想要最终把数据描绘在图上，因此存储过程数据\n",
    "b_history = [b]\n",
    "w_history = [w]\n",
    "\n",
    "# iterations\n",
    "for i in range(iteration):\n",
    "    \n",
    "    # get new b_grad and w_grad\n",
    "    b_grad,w_grad=getGrad(b,w)\n",
    "    \n",
    "    # update b and w\n",
    "    b -= lr * b_grad\n",
    "    w -= lr * w_grad\n",
    "    \n",
    "    #store parameters for plotting\n",
    "    b_history.append(b)\n",
    "    w_history.append(w)\n",
    "\n",
    "# plot the figure\n",
    "plt.contourf(x,y,Z,50,alpha=0.5,cmap=plt.get_cmap('jet'))\n",
    "plt.plot([-188.4],[2.67],'x',ms=12,markeredgewidth=3,color='orange')\n",
    "plt.plot(b_history,w_history,'o-',ms=3,lw=1.5,color='black')\n",
    "plt.xlim(-200,-100)\n",
    "plt.ylim(-5,5)\n",
    "plt.xlabel(r'$b$',fontsize=16)\n",
    "plt.ylabel(r'$w$',fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 把learning rate增大10倍尝试\n",
    "发现经过100000次的update以后，我们的参数相比之前与最终目标更接近了，但是这里有一个剧烈的震荡现象发生"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAcXElEQVR4nO3de5BcZ3nn8e8zo7vcyDehICwj5LKUGLORJWXXwjg+GIihskCFXLxZtrwbO4jdyi57oyizxlhQqQpxgSsbV9lBib2Jt/YSnAKDIRBYxJFdRkqssV0LcSKDQV7ZjoxtWVJrpJE0M+/+0d0zPaOZnr6cc97zvuf3qXKpp919+pmD6a/ePqe7zTmHiIjIfIZ8DyAiIuWmUIiISEcKhYiIdKRQiIhIRwqFiIh0tMj3AN26+OKL3fr1632PISIlNzIyAsDWrVs9T1IOIyMjrzjnVg+yDQvl9Nht27a5/fv3+x5DRErOzAAI5bktb2Y24pzbNsg2gllRvPjiWXbuPOx7jFyk6eJct58kK3PdvkgZ7dw55nsEb3buXJbp9oIJRcyS5Gyu20/T0Vy374PiJ1IchaIC8g6RD4qfSHEUCgmS4ld+Cl88FAqRkogtfr7Dl/XjVzl8CoWI5MJX+PbsyefxfYevNzqYLSJSuNhWfL3QO7NFRKSjYFYU9fo4afpy3/dPkoHemCgiUlnBhKJWM5JkuO/7DxKZslL8RKQIwYRiUINEpqwUPxEpQmVCESPFr/wUPomBQiGlEl/8jpCmE76HyIzCV00KhUjO4opfOOHrZnWq8HVHoRCRnpQ9fNNvuFt4zphe6swzegqFiFRW2aPXi5nR+5lMt61QiIhEIM/oeX9ntpmtMbMnfc8hIiJz8x4K4HPAct9DiIjI3LyGwsyuB0aBOL/jVES8OXToKd8jRMPbMQozWwLcDvwK8NA8t9kB7ABYuvT1pOmhjttMknUZTykiIWmPwwMP3MJNN93HunWbPU4UB58Hs28F7nHOHTWzOW/gnNsF7AJYu/ZnXZLUFtjkUdK0nu2UJaEIiizs4MHHpy5PTJzl4MHHFYoM+AzFu4Drzex3gM1m9ifOud8edKMLxyRMC62mQqPwSR7Wr/+FqcvDw4tn/Cz98xYK59wvti6bWZpFJGIWWwAVPslD++pBLztlpxTvo3DOJb5nkGLFFr7YXvaMIXyKRHZKEQqRGMQUvxhWfK3fIYbo+aZQiMg5Qo7e9Gc9NX6HGKIHfoOnUIhI1EKOXrvegndFpo+tUIiIBMBn8MrwER4iIlJiCoWIiHSkUIiISEcKhYiIdKRQiEiUDh36ge8RoqFQiEg02uPwwAMfVSwyolCISDQOHnxi6nLj02Of6HBr6ZZCISLRWL9+y9TlxqfHbulwa+mWQiEi0Vi37sqpyzfd9Iczfpb+KRQiEiVFIjsKhYiIdKRQiIhIRwqFiERJp8ZmR6Ho0aaVj2JMdryNMcmmlY8WNJGItOh9FPlQKHqQXHQfv/nGW3n/ms/OGwtjkvev+Sy/+cZbSS66r+AJRapN76PIh0LRpU0rHyW56H4Arlr19Tlj0YrEVau+DkBy0f1aWYgUSO+jyIdC0aVnRq/hyWO/PPXz7FjMjgTAk8d+mWdGryl8VpGq0vso8qFvuOuSY4ivvnQrwFQMWn8+/NLHed+aO8+JxFdfuhWnFotI4PQs1oNWLGavLD618TpFQqQEdDA7H3om69FcsWinSIj4037wenz8jA5mZ0QvPfXBMcTDL318xiqi5eGXPq5IVNSnPz3zeNQddzzmaZLqWrFiVdtPbtbP0i+Fog/GJO9bc+ec/+59a+7UiqKCZkdivutiUOYAnjx5rOPP0h+Fokdznd3UrnW9YiGxCimAp0+f8D1CFPRM1oP5ToH9zDN7Op46KyJ+PPbY//A9QhQUii7NF4mvvnQrkyya82woxaI6yvxyjMigFIoubVz5WMdTYOc7dXbjSj2BVIViIbHSMYouHRi9lvTVm0kuun/eU2BnvykvffVmDoxe62Nc8STWWIR0XKLdmjWX+x4hCgpFD9JXb+EfxjbyzOg18x6obsXi709cq0hINEIJ4Ne+dicjI1+Z+vmSS67wOE08FIoedfPk7xhSJEQ8eMMbNnX8WfqjYxQiEo2Z75swvY8iIwqFiERj5seML9LHjGdEoRARkY68hsLMVpnZN8zsW2b2ZTNb4nMeEQlb+4cATk5O6EMBM+J7RfEh4C7n3C8Bh4H3eJ5HRAKmb7jLh9eznpxz97T9uBr4qa9ZRCR8+oa7fPheUQBgZtuBC5xz+2Zdv8PM9pvZ/pMnj3qaTkRCpEhkx3sozOxC4G7g5tn/zjm3yzm3zTm3bcWK84sfTkSCpW+3y47vg9lLgAeBTzjnnvM5i4iET1+Fmg/fK4pbgC3AbWaWmtmNnucRkYC1n+U0MXFWZz1lxPfB7HuBe33OICLx0FlP+fC9ohARyYzOesqHQiEiUVIksqNQiIhIRwqFiIh0pFCIiEhHCoWIiHQUzDfc1euTpGm9q9smSS3naUREqiOYUNRqS0iSdV3dNk0P5TxN8RQ/kd4cOvQDnfmUkWBC0YtugxKSGOPXoghKVmZ/hIfeS5GNKEMRoxjj1xJrBBXA4s31ER4KxeAUCvEu1gjGGsB2ZYuhPsIjHwqFSE5iDWC78sXwTVOXrrzy93j22Tfx7LPdnQQzW9ki6JNCISJ9K2MM9+xp/PmBD9ww8LbKF0I/FAoRkXmUMYQ+6A13IiLSkUIhIiIdKRQiItKRQiEiIh0FczC7Xnek6cSM65Jk2NM0IiLVEUwoarVFJMnqGdel6cuepsmXAigiZRJMKOYyOxyxiDGAip9IuIIORawUwLAoghI7hUIKE2sAQRGUuCkUIhmINYIhB3D2yS9zUQi7o1CIyLxCDGDrs566mT3kEBZJoRCRygoxhD7oDXciEqVDh/b7HiEaCoWIRKM9Dg888OuKRUaCeempXjfSdLHvMfqWJGd9jyASvYMHvzd1ufFVqN9j3bptHieKQzChqNWGSZKVvscYSJqO+h4hcwqglMn69W+butz4KtS3dbi1dCuYUMQg9NDNJ8YAgiIYovbVw003PajVREYUChmYAhiWqgRQkchOV6EwM3POubyHESmTWAMI8UZQ8tHtiuJHZvZfgfudcyfyHEhE8hdzBFtvuAv55Jf5+FoNdhuKLwN3AJ82sz8G7nbOHcpvLBGRwcQYQ18rwa5C4Zz7mJndDvxL4HeA/2BmXwI+75x7PM8BW+p1SNOFx02S8QKmEREpnq/4dX0w2zl3Cvgj4I/M7J3AR4Hvmdk+4C7n3Jf7GcDM7gOuAL7unPvd+W5XqxlJsvC4adrPFOWnAIqIL90ezF4LXDDrn93AceCfAX8B9PwxjGb2QWDYObfdzO43s8udcz/sdTvtuolJiGIMoOInEoZun1WfByaBE0AdOEYjEseALzX/7EcCfLF5+VvA24GpUJjZDmAHwKpVl/b5EHGIM4CLSNM4Y6EISky6ffZ5EtgI/CnwB865gxk9/krgheblI8CW9n/pnNsF7AJYu3abTs+NUJwBBEVQYtLtweytZvYO4D8DB8zsIRoHsv9mwMc/ASxvXj4PfUihRCTWCIbyMmg3J7/MpgjOrZeD2d8FvmtmP0cjGHvMbAT4HPCVPt+QN0Lj5aZ9wM8DB/rYhogUqOwBnP7iot7nDCWCRet5Tzrn/s7MPgzcCfwXGscYngMu7+PxHwIebR4sfy9w9Xw3rJ+E9Kk+HmEByebstykiYSp7BH3p9qynERovDdWA1zH9cpE1/9zQz4M7546bWQK8G7jTOTfvQfFaDZKkn0fpLNa/QSiAIpKVbvP5BPDaAv/0xTn3GtNnPhUuj/iUgQIoIlnp9mD2h/MeRLKlAIZFAZQy0wtyEpRYAwhxRlABjINCIVISsUbQVwDzOPlltqqEMJhQ1Mcgfbq/+yZXZDuLiHSv6ADu+UpxjxvjKnAuwYSithKSeU+eXVi6L7tZykIBFPEr1lXgbMGEYlCDRKbMFEARyVtlQhGrGAMYY/xaFEEJkUIhpRNj/FpijaACGDeFQqRAsUawjAHs9+SX2RTBgEJRPw3ps76nmJZc5nsCkfIoUwD3/HHjz6xmKmMEixZMKGorILnK9xTT0id9T5APBVBkpjJF0JdgQlE2ZYpWlhRAEZlNoZAZFMCwKIBSBIVCKkEBDIsCWC7BhKJ+FtIX89l2sjaf7YrkLdYAwuARLNPJL+1CjGAwoagty/c0taxOpSsTBVBC1m8E9wx4/7yFuAoMJhR5i/VcaQVQqurQgb2s27Td9xjnKGvAOlEoIhdjAGOMHyiAWTh0YO/U5Qc+805u+tR3ShmL0CgUEpwY49cSYwSLDODBp9OpyxPjZzj4dKpQZCCYUNTHIX2l2MdMLi728URijWBRATz2+gRsCNwkNryEI69PcjsJBqqzCgwmFLWlkGwo9jHTHxf7eEVQ/MSHwgJ4xXaOfOcajjz/Q35j55dYd0X+q4kYV4GzBRMKH4oOUxFijB8ogDJtee1CVl6wppBIQLyrwHYKRcXEGD9QAEXypFBIFBTAsCiAYQkmFPVJSOu+p+hfUvM9gYQo1gBCfhF85QyMTRR/8ktLjBEMJhS1xZC80fcU/Utf8D1B9hQ/GUReETy8Ao4u8RvZ2FaCwYQidCFHrhMFUORcsa0EFQoZiAIYjsoE0DnfE0QnmFDUHaTj+T9OEswekTwpgGGZHUEz8zNIpIJ5WqwtKuYgka8DYHlS/KSlCgF8ZRxOT4R98ku7MqwE9RQyS4xnLMQYP1AAZVp7AA8vh2NL4oliGVaB+r9aBcQYP1AApRrKELxg/pM8YY5Hh8d6vt+1E8tymEbKQAEMjyIYpmD+Z6sNQbKy9/ulo73HJQQKYLxiDSAUE8FXJuF0QSe/tMQeQC+/npmtAv43MAyMAjc6587k8Vj9xCUEMQZQ8YtfERE8vBSOFXTyS0vMq0Dwt6L4EHCXc+7bZnYv8B7gq55mCVKcARwjHfU9Q/YUwPjFvAoET6Fwzt3T9uNq4KcL3WfUxtm/+FjXj7Ht7Ko+JhPfFMCwlDGCTm+4y1whoTCzLwCb2q7a7Zz7jJltBy5wzu2b5347gB0AF1x6Cdct7n7cPXQflVAofuGKM4DlfAn0iE0yZvR18ktLGQPoUyGhcM59ZPZ1ZnYhcDfwqx3utwvYBbBu2+ae/prQS1SCsXiUPWcLPEJXEAUwXGUM4AvDcLzPk1+mhb4KzDZ0vg5mLwEeBD7hnHvOxwyhijGAMa7+QAEMXRkj6IuvZ51bgC3AbWZ2G3Cvc+7PO93hFOP8YOFDGQu6ktcPvA3JVozxA7QClGj4Oph9L3BvL/c5jyG2s3zgx96bQWzKRvErrxgjWPYV4FE7y5hN9HTyS4siOLf4/iteQBaxKZ86eznle4jMKYDlVPb4/WjIOGb9zVn2CHYvgmMUkr0YAxjj6g8UwDIrewR9CWavjHGGA/j5GMVNlOBTuSooxviBAijhCSYUKxhmC+d7eewnPAUqbwqgHwpgvurudGYnv7RUPYLBhMInX4HKmwIoWSpLAEcYwmw403nKEsHuXZTp1hSKClMAw6IA+lOWCPoSTChOc4Yf8xPfY2RiA2/2PULUFMCwKIDlF0wolrOIt0byOuH3IwnebApgvhTA7pxgzOvJLy0xBTCYUMQkluDNpgBKP7IO4G4WM8ki72H1uwJcn+nWggnFWcZ4kb/PZdtr+dlctls1CmBYFMB8+Q5VloIJxTKW5LiUq3tfpuZBAcyGAihVF0wo8hbT64ktB3JagfmmAGYj1gD+dzfKGU5Hc/JLi88VYDChGOek7xGCE2P8QAGUzl7HUiZscYQhHOX7Xb+f4y2ZPnIwoVjCUs7jBD+J7G8JAKt5q+8RgqIAhkUBzI6v+AUTipY3R3gA7id83/cIuVAAe6MAZmOME7me/NJSpQAGF4oYxRg/iDOAil/vig7gSpZxhlMFPG6ZT4L5x5luLZhQTHCSozzhe4xznM8W3yOUVpwBjPPlT1AE+xHrKnC2YEKxmGW8kU2+xzjHCyWMVxYUwPnFGcB4VoBnqDPOSV5u/j4K4OCCCUVZlTFeWYgxgIpfZ7EEcAUrOMXpqd8nlgDOVmQAgwnFJKOcZF+uj7GCq3PdfkjiDOAoL3DA9xCZUwA7iyWA5+r0Mug7M32kYEKxmOUFFHR0arkaEwVwmgIYjn4D6JzLeJLyKiqCwYSiKDG+nvlyzisxXxTAaQrgtHGOMWmnSnnyS0toq8BgQuGoM0E60DaGSTKZJTQxxg8UwCroJ4BLOY/FTJY6nvkfA/xAplsLJhTDrOB8tg60jaMDhqasFMC4KIDxK3PE5hJMKLIwaGjKKsYAVjV+oAAOapLXmORU7ie/tFQhgMGEwjjO0OT/6ek+k0PvymmacokzgHWOMuJ7iFxUNYJFBXAJ32VxgY9XzhXgP890a8GEYojzWM72nu5zqsewhKAq8YNYA6gVYGxiXQG2CyYU/eg1LEGYHOUUe31PkTkFMHTlWQE6juAYG/jkl5YqR7AlnFBMHGfR8d093WX8ddfnNIxfCmA4FMDiLeJJhhnObJ4wV4AfznRrwYTCqLHIruvtTj2GJQSxxg/iDGCML39CyQOY8RvuyhJAn4IJRT96DksIIowfxBvAGOMHlHsF6F5lyM70fPJLS6kj6Ek4oThbh5fS/u+/JslqEq+ijB8ogAEqawSH+RHGyb7ni2IVOPSfMt1cOKEYqsGypP/7DxKZsookfqAAhkYBrJZwQjGoQSJTVjHGDxTAEJQ4gDb+KjZxtueTXyDuAA4inFCcqcPzafbbvSTJfptFiTF+oAAGoMwBNHse42R/M5Y4gD05//ZMN+c1FGa2Bvimc+6qBW88XINVSfZD5BEf30KOHyiAoVEAo+d7RfE5YLnXCfKIj2914Fjqe4rsKYDlVLIAvvbyS7z4D6fZ+82vsP2qVf1vKKIADspbKMzsemAUONzVHc7U4cdpniPNb0Pi53EHoQCGQwHMzN6RQ/zN/32EyUnHO//VD/jO/7yJ7VvX9bexkgWwJwP0cS6FhMLMvgAzPld3N/AO4FeAhzrcbwewA+DSn1kFq5Mcp+zAV6DytiHxPUHvYgxgjC9/gpcApvsO4iYbb7g7c3aCdN/B/kNRogD6VkgonHMfaf/ZzD4F3OOcO2pmne63C9gFsO3n1vr7fkNfgcqbAlgOMcYPvAQwuewYy5YOcebsJEsWGcllR7KfI/QVYB/Mx/fLmtkjwGTzx83AXzjnfrvTfbatr7n9twf0Vvq3JL4nqK6XU98T5GND4nuCIOx96hDp4wdJfmE92zf3uZpYSNlfAn1bOnXRzEacc9sG2ZyXUMwYwCx1ziUL3W7bxrVu/907CpgoIzG+nKD4+aUASrfesnPqYhah8H3WE91EAoCTdXgqnXnd5u7u6kWMy9O/TX1PkI9QAqiXQMOyIfE9QWa8h6JrS2twWTLzutnhiIHiVzwF0C8FMHtvyXZz4YRiLrPDEYM68Gzqe4rsKYDFUwD9iiiA4YRitA5/nWa/3X+SZL/NQcUYwBhXf6AA+qAAFi6cUCyrwRVJ9tutA0+n2W/Xt7IFMMb4gQLoQ6wBPEZ2J8Fcnc1mWsIJRZ7yCJBveay+ykABLIYC6EdJIxhOKE7UYU/qe4qFXZf4nqAhxviBAlgUBTAsOQcwnFCsqMFVie8pFnYceDL1PUX2FMB8KYDFqEoAb8h288GEwh2vM/bttKvbLnt3kussCwohaL0KYTXXDwUwXwpgMXIOYDChYGUNrk66umm3QQmN1wDGGD9QAPOmAPrxa9luLpxQ9KLLoIRGAcyBAhgWBdCLYEJxtl7nld2p7zGmXHx9UvyDKoDB0MufOVEAvQgmFHZejUW/mPgeY0qZopUVL/GDKAM4Vgf2pb7HyJwCmJOsA/hb2W4umFCcqdf5f2na8TaXJkkhswClilZWjo7D+COp7zEypwBmRwHMSckDGEwohms1zlsgBAuFJERFxg/iDGCMqz9QALMUWwCX/ftstxdMKLqxUEhCdAQ4oQAOJMb4gQKYuQgDmJVgQjFWr/PDEj5hXl7AE16MAYxx9QcKYBYUwPIJJhSLajVWl/AJs4zxmksRQetFjPGDsAJY9Mua3Yo1gEUeA7zkk9luz/tXoXbLzOrAAd9zlMTFwCu+hygJ7Ytp2hfTtC+mbXLO1QbZQDArCuDAoN/7Ggsz26990aB9MU37Ypr2xTQz2z/oNoayGEREROKlUIiISEchhWKX7wFKRPtimvbFNO2LadoX0wbeF8EczBYRET9CWlGIiIgHCoWIiHRUulCY2Soz+4aZfcvMvmxmS5rX32dme83sk223Pee62JjZGjN7tO3nDWb2HTN7ysxub1632MweNrPHzOxmf9Pma/a+aLv+YTPb3Lxc9X1xpZl9u3m5kvvCzC41s9TMdpvZLmuo6r445/fuZ1+ULhTAh4C7nHO/BBwG3mNmHwSGnXPbgQ1mdvlc13mcORdmdgHwZ8DKtqv/LfAp59xm4AYzWw38O2DEOXcN8GtmNtCba8ponn2BmX0IeNY591TzqirvCwPuAhY3r6rqvvgI8G+cc9cD64C3Ut19Mdfv3fO+KF0onHP3OOe+3fxxNfBTIAG+2LzuW8Db57kuNhPAjcDxtuteBf6Rma0BlgJHmbkvHgFifKPROfvCzC4EPg+8ZmbvaF6dUMF90fRbwHfbfk6o4L5wzt3mnPu75o8X0XiHdkIF9wVz/95zXdeR93dmm9kXgE1tV+12zn3GzLYDFzjn9pnZh4EXmv/+CLCFRjVnXxe0Dvui/WbfBD4KXALsBsY5d1+syX/afHW5L/4j8CDwBeD3mn8zquS+MLOLgH8B3ND8Byq6L9pueyPwt865F82sqvtirt+7533hPRTOuY/Mvq75N8W7gV9tXnUCWN68fB6NldBc1wVtrn0xh1uB33DOOTP7Q+DdTO+LYzT2xYn8pixGl/viKuBjzrnDZvZFqr0vPgt8wjl3tu2Joqr7AjPbAHwMeFfzqqrui7l+7573RemeXJsHrx+k8R/9c82rR5h+aenngYPzXFcFbwbWmdkyGqsoR3X3xY+ADc3L24DnqO6+uA74fTNLgc1m9rtUdF80X6v/X8DNzrljzasruS/I6LnT+4piDrfQeAK8zcxuA+4FHgIeNbO1wHuBq2k8Qc6+rgruAFIax2++RuPlp2eAvzSza4ErgL/2Nl2x7gT+pPnfyUngg8CFVHBfOOc2ti6bWeqc+6SZvYkK7gsaq+5Lgbubq6s7aBzkreK+mOv3fmGO6zoK5p3Zzb8lvBt4xDl3eL7rqqoZzLcDf9X2t6hK0r6Ypn0xrar7Yq7fu9d9EUwoRETEj9IdoxARkXJRKEREpCOFQkREOlIoRESkI4VCREQ6UihE+mRmG83Mmdl7fc8ikieFQqR/rc8Xe9zrFCI5UyhE+rcVOOice8X3ICJ5UihE+rcVeNzMdpjZ02Y22vwireA/yViknd6ZLdInM3uNxmdMPUHjM8lWAr9P43tCNjrnRj2OJ5KZMn4ooEjpmdllwPk0PlDt/a75Ny4zqwPfAK6h8YVaIsHTS08i/dna/PM2N3NZ/v3mn6sLnkckNwqFSH+20Piu7pFZ17+h+efzBc8jkhuFQqQ/W4EX57j+RhrfHLa32HFE8qNjFCL92QKMmdki59w4gJldAvxr4PPOuTNepxPJkM56EumRma0HftL853vAfwMuAXbS+Paw6xUKiYleehLpXetA9j+l8SX1DwOfBb4E3KBISGy0ohARkY60ohARkY4UChER6UihEBGRjhQKERHpSKEQEZGOFAoREelIoRARkY7+P1oI3MXHyHm4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 上图中，gradient descent最终停止的地方里最优解还差很远，\n",
    "# 由于我们是规定了iteration次数的，因此原因应该是learning rate不够大，这里把它放大10倍\n",
    "\n",
    "# y_data = b + w * x_data\n",
    "b = -120 # initial b\n",
    "w = -4 # initial w\n",
    "lr = 0.000001 # learning rate 放大10倍\n",
    "iteration = 100000 # 这里直接规定了迭代次数，而不是一直运行到b_grad和w_grad都为0(事实证明这样做不太可行)\n",
    "\n",
    "# store initial values for plotting，我们想要最终把数据描绘在图上，因此存储过程数据\n",
    "b_history = [b]\n",
    "w_history = [w]\n",
    "\n",
    "# iterations\n",
    "for i in range(iteration):\n",
    "    \n",
    "    # get new b_grad and w_grad\n",
    "    b_grad,w_grad=getGrad(b,w)\n",
    "    \n",
    "    # update b and w\n",
    "    b -= lr * b_grad\n",
    "    w -= lr * w_grad\n",
    "    \n",
    "    #store parameters for plotting\n",
    "    b_history.append(b)\n",
    "    w_history.append(w)\n",
    "\n",
    "# plot the figure\n",
    "plt.contourf(x,y,Z,50,alpha=0.5,cmap=plt.get_cmap('jet'))\n",
    "plt.plot([-188.4],[2.67],'x',ms=12,markeredgewidth=3,color='orange')\n",
    "plt.plot(b_history,w_history,'o-',ms=3,lw=1.5,color='black')\n",
    "plt.xlim(-200,-100)\n",
    "plt.ylim(-5,5)\n",
    "plt.xlabel(r'$b$',fontsize=16)\n",
    "plt.ylabel(r'$w$',fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 把learning rate再增大10倍\n",
    "发现此时learning rate太大了，参数一update，就远远超出图中标注的范围了\n",
    "\n",
    "所以我们会发现一个很严重的问题，如果learning rate变小一点，他距离最佳解还是会具有一段距离；但是如果learning rate放大，它就会直接超出范围了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAdBElEQVR4nO3dfZAkd33f8ff39p5Pw52QzuecHiyOQmcLE590V4muhFAjwEDZkDJ2WXZIJbYwR1IkfgpFiQiMQrkqWAWqxKqSzAWR5J/EhhTIyGAM5mihYB1Bh6jiyScbMQpIlkAPsze3p9M97C9/zPTt7txsz0xPd//695vPq0q1u6OZnu82Yt776+7ZNeccIiIiq1njewAREWk2hUJERHIpFCIikkuhEBGRXAqFiIjkWut7gHFdfPHF7oorrvA9hoj0HT16FIDdu3d7nqQnm+f48eMA7N271+c4jXHkyJGnnXPbp9mGhXJ57L59+9xDDz3kewwR6UuSBIA0Tb3Okcnmuf/++wEI5bWtamZ2xDm3b5ptBLOieOKJ09x225O+x6hEmq6rdPtJsqXS7ctsarcXAbjttpOeJ+nJ5sk0ZS4fbrttY6nbCyYUMUuS05VuP00XKt2+D4qfSH0UihlQdYh8UPxE6qNQSJAUP/86nbPA6nMrfPFQKEQaIrT4tdu9k8WrzV13+LJwVfX8sxw+hUJEKlF3+LJwzc9X8/xhrfh0MltEpHahrfjKpHdmi4hIrmBWFN3uGdL0x4UfnyRTvTFRRGRmBROKVstIkrnCj58mMk2l+IlIHYIJxbSmiUxTKX4iUoeZCUWMFL/mU/gkBgqFNEp88XuWND07+m6BUPhmk0IhUrG44rcUvk6nd7loU1aB2TyZceZS+MajUIjIRLLwtdu24mvfsnmW3nA3eq6mRK4MVUZPoRCRmdWUyJVhZfR+stRtKxQiIhGoMnre35ltZjvM7GHfc4iIyHDeQwF8CNjkewgRERnOayjM7EZgAYjzb5yKiETA2zkKM1sPvA/4JeDeVe5zADgAsGHDT5CmP8jdZpJcVvKUIiLi82T2LcBdzrmOmQ29g3PuIHAQYOfOn3ZJ0hqxyQ5p2i13yoZQBEXEF5+heC1wo5m9E9hjZh91zv3WtBsdHZMwjVpNhUbhEwmHt1A4516VfW5maRmRiFlsAVT4RMLRiPdROOcS3zNIvWILX2yHPRU+Wa4RoRCJQUzxG2fF1+mcHPu+dcjmyWRzKXrTUyhE5DzjRK/dXjv2feuQzbP0u556czUlZNPyGTyFQkSi1pSQTWuy4F1V6nMrFCIiAfAZvCb8Cg8REWkwhUJERHIpFCIikkuhEBGRXAqFiIjkUihERCSXQiEiIrkUChERyaVQiIhILoVCRERyKRQiIpJLoRARkVwKhYiI5FIoJrR7ywMYi7n3MRbZveWBmiYSEamWQjGB5KJ7+PVLbuHNOz64aiyMRd6844P8+iW3kFx0T80TioiUT6EY0+4tD5Bc9DEArt76maGxyCJx9dbPAJBc9DGtLEQkeArFmB5ZuI6H53/h3NeDsRiMBMDD87/AIwvX1T6riEiZ9BfuxuRYw6efugXgXAyyj/c99W7etOP28yLx6aduwanFIhI4hWICq8VieSBAkRCRuOiVbEJZLJYfhlpOkRCR2OjVrADHGu576t1D/919T71bkRCRqOgVrQBjkTftuH3ov3vTjttHvs9CRCQkCsWEhl3dtNxql86KiIRKoZjAapfAfuCR+3MvnRURCZmuehrTapHITlyvdumsTmyLSOj0CjamK7d8Jfd9EsOuhrp662e4cstXap9VRKRMCsWYji5cT/rMzcDql8AOxiJ95maOLlxf+6wiImXSoacJpM+8jX84eSWPLFy36uGkLBZ/e/x6RUJEoqBQTGicF3/HGkVCRKKhQ08iIpJLoRARkVwKhYiI5PJ6jsLMtgJ/CswBC8BNzrlTPmcSEZGVfK8o3grc4Zz7eeBJ4A2e5xERkQFeVxTOubuWfbkd+JGvWUREZDjfKwoAzGw/cKFz7vDA7QfM7CEze+jEiY6n6UREZpv3UJjZi4E7gZsH/51z7qBzbp9zbt/mzdvqH05ERPyGwszWA58A3uOce8znLCIiMpzvFcXbgGuAW80sNbObPM8jIiIDfJ/Mvhu42+cMIiKSz/eKQkREGk6hEBGRXAqFiIjkUihERCSXQiEiIrkUChERyRXMX7jrdhdJ0+5Y902SVsXTiIjMjmBC0WqtJ0kuG+u+afqDiqepn+InIr4EE4pJjBuUkMQYv4wiKNJsUYYiRjHGLxNrBBVAiYVCId7FGsFYA5jpdM6wbZteQmaB/lcWqUisAcy02xvpdE6OfZFJ1TqdMyu+nnYurQiXKBQiUti2bRsbE8R2eyMA8/O9r8uYK/ZV4bgUChGRVTQlgr7pDXciIpJLoRARkVwKhYiI5FIoREQkVzAns7tdR5qeXXFbksx5mkZEZHYEE4pWay1Jsn3FbWn6Y0/TVEsBFJEmCSYUwwyGIxYxBlDxEwlX0KGIlQIYFkVQYqdQSG1iDSAoghI3hUKkBLFGMC+AnY7r3+fsqvepUzZPZpy5FMLxKBQisqq8ALbb60bep07ZPEu/62n0XLGuBMumUIjIzGpK5JpOb7gTEZFcCoWIiOQK5tBTt2uk6TrfYxSWJKd9jyAiUkgwoWi15kiSLb7HmEqaLvgeoXQKoEj8gglFDEIP3WpiDCAogiIZhUKmpgCGRQGUSY0VCjMz55wbfU+ReMQaQCgngp2O9bfVjHOH2TxSvnFXFH9vZv8F+Jhz7niVA4lI9cqIYLs9V9q2ypDNk73hrikBK5Ov1eC4ofgU8H7gP5rZfwXudM79oLqxRESm05SAlcnX4dCxQuGce5eZvQ/4V8A7gd81s08CH3bOfa3KATPdLqTp6HGT5EwN04iI1M9X/MY+me2cex74E+BPzOw1wG8Df2Nmh4E7nHOfKjKAmd0DXAV8xjn3h6vdr9UykmT0uGlaZIrmUwBFxJdxT2bvBC4c+OcQcAz4NeB/AxP/GkYzewsw55zbb2YfM7OXOef+btLtLDdOTEIUYwAVP5EwjPuq+kNgETgOdIF5epGYBz7Z/1hEAny8//nngVcC50JhZgeAAwBbt15e8CniEGcA15KmccZCEZSYjPvq8zBwJfDfgf/snGuX9PxbgMf7nz8LXLP8XzrnDgIHAXbu3KfLcyMUZwBBEZSYjHsye6+ZvRr498BRM7uX3ons/zvl8x8HNvU/vwD9kkKJSKwRzA6DLr2Pohnf5+D7KIrMpQgON8nJ7C8BXzKzn6EXjPvN7AjwIeDPC74h7wi9w02HgZ8DjhbYhojUKAtgu20rvvYtm2fpDxdNPleM5wLLMPGedM5918zeDtwO/Ad65xgeA15W4PnvBR7onyx/I3DtanfsnoD0GwWeYYRkT/nbFJEwNSV6TTPuVU9H6B0aagEvYulwUbbW21XkyZ1zx8wsAV4H3O6cW/WkeKsFSVLkWfLF+hOEAigiZRk3n18HnhvxTyHOuedYuvKpdlXEpwkUQBEpy7gns99e9SBSLgUwLAqgNJkOyElQYg0ghBfBTv/Xg+adO1QA46BQiDREaBFsf6v3cdTcdQWwM/B7rau4+GXQrIQwmFB0T0L6nWKPTa4qdxYRGV9dAczCNf9Mfc8b2iqwqGBC0doCyaoXz46WHi5vlqZQAEX8Cm0VWFQwoZjWNJFpMgVQRKo2M6GIVYwBjDF+GUVQQqRQSOPEGL9MTBHsnOh9TL+jAMZOoRCpUUwRbH+59zG5thkBzMKVKXrxyyBFMKBQdF+A9Hu+p1iSvNT3BCLN0YQAZuGa/4fex7JmakIEfQsmFK3NkFzte4ol6cO+J6iGAiiyUhMi6FswoWiaJkWrTAqgiAxSKGQFBTAsCqDUQaGQmaAAlq/zfH+GCs4dKoDNEkwouqchfaKabSc7q9muSNV8BrB9b7UzTBrBLFznHt+gi1+WCzGCwYSitbHay9TKupSuSRRACdmkAcrClf31s6auIkM8DBpMKKoW67XSCqBIszQ1YHkUisjFGMAY4wcKoDSXQiHBiTF+mZAi2DnV+zjq3KECGL5gQtE9A+nT9T5ncnG9zycSUgTbm3sfx5m5jgBm4Tr3nBVd/LLcrEQwmFC0NkCyq97nTB+t9/nqoPiJD3UEMAvXuZPZNUU3pFVgUcGEwoe6w1SHGOMHCqD4E9IqsCiFYsbEGD9QAEWqpFBIFBTA+nVO9z4WOXeoAIYlmFB0FyHt+p6iuKTlewIJUZMD2N7U+1h0xrIjmIXr3PZrvvglE2MEgwlFax0kl/ieorj0cd8TlE/xk2mUHcEsXOdOZnuMbJNXgkUEE4rQhRy5PAqgyPmavBIsQqGQqSiA4VAApahgQtF1kJ6p/nmSYPaIVEkBHK3T//9jE84dKoLVCuZlsbW2npNEvk6AVUnxk0yZAWxvKH+bRaWPL4Xr3G0NCFgZmhBBvYQMiPGKhRjjBwqgLEkuWQrX/LLbYtCEw6D6v9oMiDF+oADKbGhC8IL5T/K4OR6YOznx464/u7GCaaQJFEC/Oq73cZJzh4pgmIL5n621BpItkz8uXZg8LiFQAOMVSgDb63ofJ5m3yghm4Tr3XDVc/JKJPYBevj0z2wr8KTAHLAA3OedO5T+qmCJxCUGMAVT84ldlBLNwnTtHUWNwQ1kFFuWrg28F7nDOfcHM7gbeAHza0yxBijOAJ0kXfM9QPgUwfqGsAovyEgrn3F3LvtwO/GjUYxbsDA+tmx91t3P2nd5aYDLxTQEMx7wtstWt8T2G1KCWUJjZR4Ddy2465Jz7gJntBy50zh1e5XEHgAMAF15+KTesG3/c+xk/KqFQ/MIVYwAfnYPO2cVCF5lUYd4WV3w9zVxaBa5USyicc+8YvM3MXgzcCfxyzuMOAgcBLtu3x612v2EmiUow1i1w/+kaz9DVRAEM17a55kTw0bnex07/6+nmCn0VWG7ofJ3MXg98AniPc+4xHzOEKsYAxrj6AwUwdE0JYBP4etV5G3ANcKuZ3Qrc7Zz7s7wHPM8ZvjX6VMZIP8tPTL0NKVeM8QO0ApRo+DqZfTdw9ySPuYA17GfT1M/9YAmxaRrFr7lijGC2AuxaL4KTXGRSpWyeTJG5FMHh4vuveIQyYtM8XR7ked9DlE4BbKYsft82W/G1b9k8z/S/LjJXPIdBIzhHIeWLMYAxrv5AAWyypkSvaYLZKyc5xVH8/BrF3TTgt3LNoBjjB/EEcIHeH6nOzh0qgPEKJhSbmeMatnl57q97ClTVFEA/YgngV+i92S77fnwHMAtXpoyLXzKzHsFgQuGTr0BVTQGUMvkOYBauJ/tflzmP7whO7qJSt6ZQzDAFMCwKoD++I+hbMKF4gVM8yvd9j1GKXbzE9whRUwDrcYIXAKY+d6gANl8wodjEWl4RyXHCb0YSvEEKYLWaFsDP9l8+pp2rrABm4cr4uvglE1MAgwlFTGIJ3iAFUIooK4CfHXg58x1WvyvAK0rdWjChOM1JnuBvK9n2Tn66ku3OGgUwLApgtXyHqkzBhGIj6ytcynW9L1OroACWQwEc7nl6v8a7KecOs3mkfMGEomoxHU/MHK1oBeabAliOaQO4hfWlbKcs2TyZpgSsLD5XgMGE4gwnfI8QnBjjBwqgjKcpASvPAt8c+/0cLy/1mYMJxXo2cAHH+X5kPyUAbOcVvkcIigLYDKf6P7yNOneoAJbHV/yCCUXmJRGegPs+3/Q9QiUUwMmEFsDN/UM9o+auK4CnBo46VHXxS2aWAhhcKGIUY/wgzgAqfpOrK4CbB85RVP+8Tb4I5p+UurVgQnGWE3T4uu8xzrONa3yP0FhxBjDOw5+gCBYR2iqwqGBCsY6NXMJu32Oc5/EGxqsMCuDq4gzg5CvAUywA8OOGrByzeTLZXArg9IIJRVM1MV5liDGAil++SQO4qf9X1JoSzk0Df9UtmyvGQ6BQbwCDCcUiC5zgcKXPsZlrK91+SOIM4AKPc9T3EKVTAPM1JWTlyzsM+ppSnymYUKxjUw0FXWjMMrpMCuASBbA8ZzgOUNm5QwVwtLoiGEwo6hLj8cwfV7wS80UBXOIjgOvZXPFzTxbALFyZJl78kgktgsGEwtHlLOlU25gjKWWW0MQYP1AAZ8EkEcrCVeSxdav+HOA/K3VrwYRijs1sY+9U2+hMGZqmUgDjEkoAFzkGMPa5QwVwSZMjNkwwoSjDtKFpqhgDOKvxg3ACuI6/Asaft+oAZuHKVH3xS2YWAhhMKIxjrFn864kes7jmtRVN0yxxBrBLhyO+h6jErEaw6gBm4arr+TLNXAH+81K3Fkwo1nABm9g/0WOenzAsIZiV+EGsAYxnBejoAHCWdGbjB+GsAKcRTCiKmDQsQVhc4Hke9D1F6RTA8Kzlq0D2/fhfAWbhykx78UtmliOYCScUZ4+x9tihiR5y5kU3VjSMXwpgOBTA+mThov83G8qaJ8wV4NtL3VowoTBarLUbJnvQhGEJQazxgzgDGOPhT1AAZ00woShi4rCEIML4QbwBjDF+ACwuYDwHMPFFJlXJ5skUnWuWIjiucEJxugtPpcUfvyMpaxKvoowfKIABWsOLgObEcA3f7X/2LFB8rihWgWt+v9TNhROKNS3YmBR//DSRaapI4gcKYGhiDmBTwtck4YRiWtNEpqlijB8ogCE4dgg70zvUM+lFJlXJ5skUmSvmAE4jnFCc6sIP0/K3e2lS/jbrEmP8QAEMwFq7Aev/iuumxNDO/crteaDgXA2J3tS2va/UzXkNhZntAD7nnLt65J3nWrA1KX+IKuLjW8jxAwUwFKc7o+8TmKZEr2l8ryg+BGzyOkEV8fGtC8ynvqconwLYLGvavY9NCeBAuB783J+z/+qtxbcX0QpwWt5CYWY3AgvAk2M94FQXHk2rHGl1uxI/zzsNBTAcCmA51rSZ775AdujpNb/xLb74P/8l+/deVmx7TQlgEVP0cZhaQmFmH4EVv1f3EPBq4JeAe3MedwA4AHD5T26F7UmFU+bwFaiq7Up8TzC5GAMY6uHPF/o/wa82v4cAzh87ee7zU6fPkh5uFw9FUwLYALWEwjn3juVfm9kfAHc55zpmlve4g8BBgH0/s9NVOmQeX4GqmgLYDKHGb67d+7ja/HUH8IUOWzedOffl+rVG8tJny58j9BVgAb4OPb0WuNHM3gnsMbOPOud+K/cRz3fh22kds5Xj5YnvCUZTAMOyK/E9wWTqDuBcm60vgt4RbfjiPb/B/j0FVxN5QjgEenm5m/MSCufcq7LPzSwdGQmA9a2wSh5S1MYVQvxAAazLyf6hp2nn2pVMOchwlUQiE+oqsCDfVz3hnEvGuuOJLnwjXXnbnvEe6kVIURtXjPEDBbCode3ex2nnKiuAJwcu1/Ud1l2J3+cvkfdQjG1DC16arLxtMBwxUPzqpwD6VVYAs3D1r3ryHlafoXp5uZsLJxTDDIYjBl3ge6nvKcqnANav6gAudOp5nkGzFsAGCCcUC134alr+dv9pUv42pxVjAGNc/cFsB3BDu57nGbRamBYGDj2FtlJscADDCcXGFlyVlL/dLvCdtPzt+ta0AMYYP1AAfVgtTBva/U/m8+/XVPOUdynvteVsJhNOKKpURYB8q2L11QQKYD3GCeDxzvj3rcPxgRXFsLmaHEBobNzCCcXxLtyf+p5itBsS3xP0xBg/UADrMk4AN7XHv28dsnmyFcWwuZoStbJVHMBwQrG5BVcnvqcY7RjwcOp7ivIpgNUKMYDH+j/B580eYgBDNBjA15e7+WBC4Y51OfmFdKz7bnxdUuksI4UQtEmFsJorQgEsbku79zFv9joDeGzg0FOVzz1jAQwmFGxpwbXJWHcdNyih8RrAGOMHCmDV6gxgFq7s0FOVz930FeCvlLu5cEIxiTGDEhoFsAIKYHGdTn3PlZnFADZAMKE43e3y9KHU9xjnXHxjUv+TKoDBmInDn/+nXd9zZfKi1Bk49BTSarEpAVxFMKGwC1qsfVXie4xzmhStsniJH0QZwJNd4HDqe4zSNTqAWbiemh9936YpO2q/We7mggnFqW6X/5emufe5PElqmQVoVLTK0jkDZ76c+h6jdApgeZYHcPG53k/wTVkRZvNkJpmr0QFsgGBCMddqccGIEIwKSYjqjB/EGcAYV3/QgAB+u73ya9+yeZ7trygmmCu2FeDG3yl3e8GEYhyjQhKiZ4HjCuBUYowf+A/g6f45gbLn8B5AOU8woTjZ7fJ3DXzBfFkNL3gxBjDG1R/MVgDt0XYlcxQNz+mBk9m+QzrIWwBLEEwo1rZabG/gC2YT4zVMHUGbRIzxg7ACWPdhzXEVDU8WLjrzU22nKnWeA7z0veVuz5xz5W6xImbWBY76nqMhLgae9j1EQ2hfLNG+WKJ9sWS3c641zQaCWVEAR51z+3wP0QRm9pD2RY/2xRLtiyXaF0vM7KFpt7GmjEFERCReCoWIiOQKKRQHfQ/QINoXS7QvlmhfLNG+WDL1vgjmZLaIiPgR0opCREQ8UChERCRX40JhZlvN7C/N7PNm9ikzW9+//R4ze9DM3rvsvufdFhsz22FmDyz7epeZfdHMvmFm7+vfts7M7jOzr5jZzf6mrdbgvlh2+31mtqf/+azvi581sy/0P5/JfWFml5tZamaHzOyg9czqvjjv+y6yLxoXCuCtwB3OuZ8HngTeYGZvAeacc/uBXWb2smG3eZy5EmZ2IfA/gC3Lbv63wB845/YArzez7cC/A444564DfsXMpnpzTROtsi8ws7cC33POfaN/0yzvCwPuANb1b5rVffEO4N84524ELgNewezui2Hf98T7onGhcM7d5Zz7Qv/L7cCPgAT4eP+2zwOvXOW22JwFbgKOLbvtGeAfm9kOYAPQYeW++DIQ4xuNztsXZvZi4MPAc2b26v7NCTO4L/p+E/jSsq8TZnBfOOdudc59t//lRfTeoZ0wg/uC4d/3sNtyeX9ntpl9BNi97KZDzrkPmNl+4ELn3GEzezvweP/fPwtcQ6+ag7cFLWdfLL/b54DfBi4FDgFnOH9f7Kh+2mqNuS9+D/gE8BHgP/V/MprJfWFmFwH/Anh9/x+Y0X2x7L43Ad92zj1hZrO6L4Z93xPvC++hcM69Y/C2/k+KdwK/3L/pOLCp//kF9FZCw24L2rB9McQtwK8655yZ/THwOpb2xTy9fXG8uinrMea+uBp4l3PuSTP7OLO9Lz4IvMc5d3rZC8Ws7gvMbBfwLuC1/ZtmdV8M+74n3heNe3Htn7z+BL3/6B/r33yEpUNLPwe0V7ltFrwEuMzMNtJbRTlmd1/8PbCr//k+4DFmd1/cAPyRmaXAHjP7Q2Z0X/SP1f8v4GbnXP+vGM3mvqCk107vK4oh3kbvBfBWM7sVuBu4F3jAzHYCbwSupfcCOXjbLHg/kNI7f/MX9A4/PQJ81syuB64CvuptunrdDny0/9/JCeAtwIuZwX3hnLsy+9zMUufce83sp5jBfUFv1X05cGd/dfV+eid5Z3FfDPu+Hx9yW65g3pnd/ynhdcCXnXNPrnbbrOoH85XAXy37KWomaV8s0b5YMqv7Ytj3Pem+CCYUIiLiR+POUYiISLMoFCIikkuhEBGRXAqFiIjkUihERCSXQiFSkJldaWbOzN7oexaRKikUIsVlv1/sa16nEKmYQiFS3F6g7Zx72vcgIlVSKESK2wt8zcwOmNl3zGyh/4e0gv9NxiLL6Z3ZIgWZ2XP0fsfU1+n9TrItwB/R+zshVzrnFjyOJ1KaJv5SQJHGM7OXAtvo/UK1N7v+T1xm1gX+EriO3h/UEgmeDj2JFLO3//FWt3JZ/s3+x+01zyNSGYVCpJhr6P2t7iMDt/+j/scf1jyPSGUUCpFi9gJPDLn9Jnp/OezBescRqY7OUYgUcw1w0szWOufOAJjZpcC/Bj7snDvldTqREumqJ5EJmdkVwPf7//wN8N+AS4Hb6P31sBsVComJDj2JTC47kf2L9P5I/X3AB4FPAq9XJCQ2WlGIiEgurShERCSXQiEiIrkUChERyaVQiIhILoVCRERyKRQiIpJLoRARkVz/Hx0x4nqQlnUSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 上图中，gradient descent最终停止的地方里最优解还是有一点远，\n",
    "# 由于我们是规定了iteration次数的，因此原因应该是learning rate还是不够大，这里再把它放大10倍\n",
    "\n",
    "# y_data = b + w * x_data\n",
    "b = -120 # initial b\n",
    "w = -4 # initial w\n",
    "lr = 0.00001 # learning rate 放大10倍\n",
    "iteration = 100000 # 这里直接规定了迭代次数，而不是一直运行到b_grad和w_grad都为0(事实证明这样做不太可行)\n",
    "\n",
    "# store initial values for plotting，我们想要最终把数据描绘在图上，因此存储过程数据\n",
    "b_history = [b]\n",
    "w_history = [w]\n",
    "\n",
    "# iterations\n",
    "for i in range(iteration):\n",
    "    \n",
    "    # get new b_grad and w_grad\n",
    "    b_grad,w_grad=getGrad(b,w)\n",
    "    \n",
    "    # update b and w\n",
    "    b -= lr * b_grad\n",
    "    w -= lr * w_grad\n",
    "    \n",
    "    #store parameters for plotting\n",
    "    b_history.append(b)\n",
    "    w_history.append(w)\n",
    "\n",
    "# plot the figure\n",
    "plt.contourf(x,y,Z,50,alpha=0.5,cmap=plt.get_cmap('jet'))\n",
    "plt.plot([-188.4],[2.67],'x',ms=12,markeredgewidth=3,color='orange')\n",
    "plt.plot(b_history,w_history,'o-',ms=3,lw=1.5,color='black')\n",
    "plt.xlim(-200,-100)\n",
    "plt.ylim(-5,5)\n",
    "plt.xlabel(r'$b$',fontsize=16)\n",
    "plt.ylabel(r'$w$',fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 这个问题明明很简单，可是只有两个参数b和w，gradient descent搞半天都搞不定，那以后做neural network有数百万个参数的时候，要怎么办呢\n",
    "这个就是**一室不治何以国家为**的概念"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 于是这里就要放大招了！！！——Adagrad\n",
    "我们给b和w客制化的learning rate，让它们两个的learning rate不一样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the function will be y_data=-188.3668387495323+2.6692640713379903*x_data\n",
      "error 0 is: 73.84441736270833 \n",
      "error 1 is: 67.4980970060185 \n",
      "error 2 is: 68.15177664932844 \n",
      "error 3 is: 28.8291759825683 \n",
      "error 4 is: 13.113158627146447 \n",
      "error 5 is: 148.63523696608252 \n",
      "error 6 is: 96.43143001996799 \n",
      "error 7 is: 94.21099446925288 \n",
      "error 8 is: 140.84008808876973 \n",
      "error 9 is: 161.7928115187101 \n",
      "the average error is 89.33471866905532\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de5ScdZ3n8fe3Ox1ya0NIQmKgoRMMzUUhJFFpNPJwUfA6qzNnWMdddXAER3fdGdedQRFBZ85Z9ahnZzmHS2ZkHM+Z9bqKF1RkggVZCA4JRMGQDrk0aQi3JJJUOoSku3/7R1enO53u6qrq56nf8/vV53VOTroeup765gH6nV89T1WZcw4REZHxNPkeQERE8k2hEBGRshQKEREpS6EQEZGyFAoRESlriu8BKjVv3jzX3t7uewwRkaBs2LBht3Nu/mT2EUwo2tvbWb9+ve8xRESCYmZPTXYfwYRi164j3HTTc77HyESh0JLp/pNkZqb7F5F8uemmaanuL5hQxCxJjmS6/0KhN9P9+6D4idSPQtEAsg6RD4qfSP0oFBIkxS//FL54KBQiORFb/BS+eCgUIpIJhc8nncwWEam72MJXDb0yW0REygpmRVEs9lEovFjz/ZNkUi9MFBFpWMGEorXVSJLmmu8/mcjkleInIvUQTCgmazKRySvFT0TqoWFCESPFL/8UPomBQiG5El/89lIo9PseIjUKX2NSKEQyFlf8FL5GpFCISFViCl9MT3VmGT2FQkQaVrzRW5jqvhUKEZEIZBk976/MNrMFZvao7zlERGRs3kMBfBWY7nsIEREZm9dQmNmlQC8Q52eciohEwNs5CjObCtwAvBe4c5zvuQa4BuCEE06mUOgpu88kaUt5ShER8Xky+zrgFufcS2Y25jc451YDqwEWLTrLJUnrBLt8iUKhmO6UOaEIiogvPkNxOXCpmX0CWGZm/+Sc+4vJ7nTimIRpotVUaBQ+kXB4C4Vz7i1DX5tZIY1IxCy2ACp8IuHIxesonHOJ7xmkvmILX2xPeyp8MlIuQiESg5jiF9OKT9GbPIVCRI6j6OWPz+ApFCIStViiV13wzkn1sRUKEZEA+AxeHt7CQ0REckyhEBGRshQKEREpS6EQEZGyFAoRESlLoRARkbIUChERKUuhEBGRshQKEREpS6EQEZGyFAoRESlLoRARkbIUChERKUuhqFLHzLUYA2W/xxigY+baOk0kIpIthaIKydxv8P5TruM9C740biyMAd6z4Eu8/5TrSOZ+o84TioikT6GoUMfMtSRz7wDggtl3jRmLoUhcMPsuAJK5d2hlISLBUygqtKX3TTy6751Hb4+OxehIADy6751s6X1T3WcVEUmTPuGuQo4mfvL8dQBHYzD0+0+f/xveveArx0XiJ89fh1OLRSRwCkUVxorF8o/fBdzFjcd9912lX+m68cYHUt+niEg5CkWVRsZiMBL19YUvxPdUluInkm8KRQ0cTfz0+b8hixVDI4oxfqAASjwUihoYA7x7wVfGeLpJZFiMAVT8GpNCUaWRVze5fwX7gO+JROonxviBAjgRhaIKY10C+8gt78z8qqdY/+cUyYtK/x+7+urbaWt7bcbT5I9CUaHxXicxFIPxLp1NIxYx/m1H8ZMQ3XHHtQ0ZC4WiQmfOfKDsimG8WGw+sIqu3lX1HzjnYowfKICN4Jvf/Dg33HC/7zHqSqGoUFfvKgp7riaZe8e4TyuNjkVhz9WKRIOJMYBf/vKVHDpU9D1GbgwM9Pseoe4UiioU9nyEZw+dyZbeN437dNJQLLSSkFj87d/+0vcImdDqr3IKRZUq+eHvaFIkRHKuktXfbbd9iOef33rMtqamxvuxqTciEhEZx+7dO4/b1tl5lYdJ/FIoRETGYWajbjdx+eUf9zSNPwqFiMg4mpunHnN76tSZnibxy+uTbWY2G/gO0Az0Alc55w77nElEZMjMmbN55ZXiMbcbke8VxQeArzvn3gY8B1zpeR4RkaMGBvrK3m4UXlcUzrlbRtycD7zgaxYRkdGcK3+7UfheUQBgZp3AHOfcQ6O2X2Nm681s/cGDL3maTkQaVUvLCcfcnj17oadJ/PIeCjM7CbgZuHr0P3POrXbOrXTOrZwx48T6DyciDaun53F2737qmG3z55/uaRq/vIbCzKYC3wc+45x7aqLvFxGplwcf/NdRW4zzz3+Hl1l8872i+AiwHLjezApm1nivZBGRXBr9YrvZsxc03LvGDvF9MvtW4FafM4iIjGXu3DZ27+4+envhwqX+hvHM94pCRCSXli7tLHu7kSgUIiJjePbZrlG3t3iaxD+FQkRkDKPDcODAXk+T+KdQiIiM0tPzOLt2bT5m26xZczxN459CISIySnf3I8Dwy7DNmhr20lhQKEREjjNjxrFv/nfRRe9v2EtjIaBPuCsWBygUKvvc3iRpzXgaEYnZ6PMThw71epokH4IJRWvrVJKkraLvLRR6Mp6m/hQ/kXoa/e5/DfpugCXBhKIalQYlJDHGb4giKHmzcOFrjrn96ld3eJokH6IMRYxijN+QWCOoAIbrqad+N+KWcfDgPm+z5IFCId7FGsFYAzhSjDHs6XmcTZvuPXq7uXkK7e3LPU7kn0IhkpFYAzhSjDHcuXMdAwP9R28vW/aOhr7iCRQKEZmEGGO4YUM7O3YM396///SKr7iMlUIhIjLCsZ+mabS1NbFqVXxBrIZecCciMsKxn6bp0KdrKhQiIsfo7d0z4paNWmE0JoVCRGQE50a+uE4rClAoRESOsXfvyI9AbdKKgoBOZheLjkKh/5htSdLsaRoRiVVf3yvA4DvGNjdPpb399Z4n8i+YULS2TiFJ5h+zrVB40dM02VIARfzo6dnIE0/8GzAYiiuvvI62tmWep/IvmFCMZXQ4YhFjABU/CUF398M4N/jMhXNOTzuVBB2KWCmAYVEE4zHyaaampmY97VSiUEjdxBpAUARjceTIId8j5JJCIZKCWCMYawCHjA7h+vXfPfr1wEA/3d0P6xwFCoWIlBFrAIeMDOG+fb/liSfWjPinzezdu+K4qy0bkUIhIg1rZAjXrt0EDJRuGcuXX8W73nW5l7nyRi+4ExEB2tsvwswAmDLlBM4//089T5QfWlGIiABtbSuZNWshfX0vc9lln6WtbaXvkXIjmFAUi0ah0OJ7jJolyRHfI4hIGTt3PkyxuAswfvnLGzj55LMUi5JgQtHa2kySzPQ9xqQUCr2+R0idAiix2LLl7tJXjv7+I3R3P6hQlAQTihiEHrrxxBhAUAQbTV/f0L9vo7m5hfb2i7zOkycKhUyaAhgWBfB4PT3refjhfwagqamJK6/8O60mRqgoFGZm7tg3aReJXqwBhHgjCLWFsLv7QQYG+gBwDg4e3Jv2WEGrdEWx1cz+AbjDOXcgy4FEJHuK4LH27XsL8GXAYdbC3r1vyeXFM75Wg5WG4kfAjcAXzOwfgZudcz3ZjSUiUpvaIpiwefNspkyZSpLcwIoVSdpjpcLXSrCiUDjnPm1mNwAfAj4B/JWZ/RD4mnPu4SwHHFIsQqEw8bhJ0leHaUQkJjt23MehQ39g8NLY/8HJJ59LW9uFvsc6jq+VYMUns51zLwO3AbeZ2WXAJ4EHzewh4OvOuR/VMoCZfQM4B7jLOff3431fa6uRJBOPWyjUMkX+KYAi2dm8+Selrxz9/Yfp7l6by1D4UunJ7EXAnFG/7gX2A/8R+AFQ9fsRm9n7gGbnXKeZ3WFmS51zT1a7n5EqiUmIYgyg4id5MXPm4Hs+mTWXPv50leeJ8qXSn6pPM/huWQeAIrCPwUjsA35Y+r0WCfC90te/At4MHA2FmV0DXAMwe/ZpNT5EHOIM4BQKhThjoQiGZd++wVOu55//n1m+/ENaTYxS6U+fR4EzgW8C/8s5153S488Enil9vRdYPvIfOudWA6sBFi1aqctzIxRnAEERDEdPz0M8+ui/APD4499l+fIPeZ4ofyo9mb3CzC4B/jvQZWZ3Mngi+98n+fgHgOmlr2ehd7OViMQawdieBt2584Gjn5Ot8xNjq+Zk9q+BX5vZ2QwG4z4z2wB8FfhxjS/I28Dg000PAecDXTXsQ0TqKLYA7tx5CTt2ABhmU9m7N6noCstGUvXRcM49YWYfBb4CfJbBcwxPAUtrePw7gbWlk+VvB8bNePEgFDbW8AgTSPQphyINbc6cxQAsXfpOVq36LG1tnZ4nyp9Kr3rawOBTQ63Aqxh+ushKvy+p5cGdc/vNLAHeCnzFOTfuSfHWVkiSWh6lvNiW0UMUQJHKPPHEDwFYsuRyRWIcla4oHgH+MMGvmjjn/sDwlU91l0V88kABFJlYT8867r77rwFYs+Y6TjnlDYrFGCo9mf3RrAeRdCmAYVEA/ejuLhx9M8DBz6AoKBRj0BkbCUqsAYQ4I5j3ALa3J5gZzrnSC+0S3yPlkkIhkhOxRjDfAeykZepCWqbOZemy29m2p5Nteyq/d95DmJZgQlE8BIVNtd03OSfdWUSkcnkOYF/fK9z342dZ1NbB8uXQ1l7d/fMdwfQEE4rWmZBM4jUwhYfSmyUvFECRydm08QeAo3vrfXzrlsv44MfX0NZe+TmKPEcwTcGEYrImE5k8UwBFard18y9KXw0Mvip7a6GqUDSKhglFrGIMYIzxG6II5svUE1qBEe8a+5rE70A5pVBI7sQYvyGxRjDUAPYdPsiMmfO4MPkU7a9JtJoYh0IhUkexRjDUAG7f2cXUV72O/kWfYdtB2DbGBTOhRjBNwYSi+AoUtvmeYlhyhu8JRPIjxAA651j3rd8zb9E5nDFvHW2vGXs1EWoE0xRMKFpnQHKB7ymGFR71PUE2FEBpFFsf+zmHXznArh0P862vXsYHP71mzFiEGMG0BROKvMlTtNKkAEqj2PzoqM/J7iqMu6podAqFHEMBDIsCWLvpM08CRlzx1JH4HSjHFAppCApgWOoRwIGBPpqaW7j4PTey+KxLtZooI5hQFI9AYVc2+04WZbNfkazFGkDIPoJd2zYz/aQOBs6+nm3AtjpdLBPiKjCYULROy/YytVrfRyrPFEAJWdYRfOT2jcyYPpszZqyjraN+q4kQV4HBhCJrsV4rrQCKHO+p399Pcc/TFHmGb33xMj74+TV1i0WIq0CFInIxBjDG+IECWE+bH76z9JWjv+8w3ZsKdV1VhEahkODEGL8hMUYwjwGceeICoHTF05SptJ+T+B0o54IJRbEPCrvr+5jJvPo+nkisEcxbAHcWDYDT3nY9c86+km2tnWyr4WKZPEYwC8GEovUESJbU9zEL2+v7ePWg+IkPeQvgj3++mZlzFvDhT39h0vvKWwSzEEwofKh3mOohxviBAijV2dPTxbxTO1LZV94imAWFosHEGD9QAKU6u5/u4uw3v8/3GMFQKCQKCmBYfAbw4P49vLx/D/tffJqeTetoO0dXO00kmFAUB6BQ9D1F7ZJW3xNIiGINIPiL4K41/xeArevvZvtvC5x3/Rpmn5leLGJcBQYTitYWSE7xPUXtCs/4niB9ip9Mhq8Ifm/1rwa/cAO4/sOc9FyBVVemu6qIbSUYTChCF3LkylEAJTRTpk0HwJpKr6FYkaT+GLGtBBUKmRQFMBwK4KDDvUVmLzydFe+9lvYVCW2v0zmKiQQTiqKDQl/2j5MEc0QkSwpgWKqJ4J6dXby64wJWffgz2Q0UmWB+LLZOqc9Jonq/+rseFD8Z0ugBdP197Hl6G9Pf+EfBXByTh5WgfoSMEuMVCzHGDxRAGVZpAPd07+D+viNccF4HFwQSzTysAvW/WgOIMX6gAEr19uzoAmDe4rM8T1K5PKwCg/lP8oA51jYfqvp+q/qnZTCN5IECGB7fEdz6wN0AvHJgv99BAhNMKFqbIJlZ/f0KvdXHJQQKYLxiDSD4jeC+jevY+J1bAfj2J9/Lef+4htnL0rniyXcAs+blj2dms4HvAM1AL3CVc+5wFo9VS1xCEGMAFb/4+Yzg2k0F6O8HwB05zEmbCqy6PJ1QxLwKBH8rig8AX3fO3WNmtwJXAj/xNEuQ4gzgIQq9vmdInwKYD+0XJYNfmNHcMnX4dgpiXgWCp1A4524ZcXM+8MJE9+m1Pta37Kv4MVYemV3DZOKbAhiWkCI4f+nZAJyRXMHFn/o8bSv1QrtK1SUUZnY7MPLN3+91zn3RzDqBOc65h8a53zXANQBzTjuVi1sqH/c+Ko9KKBS/cMUZwLCeAt2343cAzPjw1XS/8QK6GX/2kAJYD3UJhXPu2tHbzOwk4Gbgj8vcbzWwGqBt5TJXzWNWE5VgtPRy35E6vDy9zhTAcIUUwI09T7IBuOS8DuZNOHfoq8B0Q+frZPZU4PvAZ5xzT/mYIVQxBjDG1R8ogHmz+8ktNE2ZwpzFiyv6/pAimDVfP3U+AiwHrjez64FbnXPfLXeHl+nj8YlPZUzotZw86X1IumKMH6AVYM7s3tLFnMWLaW5p8T1KcHydzL4VuLWa+8yiiU6mT/qx16UQm7xR/PIrxgiGugLseXIzM85cXPaimFAjmLX4/iueQBqxyZ8i63jZ9xCpUwDzKcT4DfT3c9eT25k3dw6nr3+U9s7Xj/l9oUbweBGco5D0xRjAGFd/oAD68NiP7qL/yBF2/L/fcNtl7+Vja340ZixCjGA9BHNUDnGYLvy8jWIHOXhXrgYUY/xAAfRh08+GPv7U0Xf4CNsKD4y7qpDjBROKGTSznBO9PPYjngKVNQXQDwWw/npPHDyBbc1NNE1twZJzqro4Js8RrIdgQuGTr0BlTQGUNOU5gNsO9bO9dSZvv+4v6bikkzM6l1d1/zxHcGxzU92bQtHAFMCwKIC1e65rO6eceybv+Ownarp/niNYD8GE4hUOs50dvsdIxRIqe8GP1EYBDEs9Avh813bOeduqzB8nVsGEYjpTeF0kzxM+FknwRlMAs6UA1uaV/b3se/YFmjvm1fWCmJhWgMGEIiaxBG80BVBqkXUAt295GoCVHWfVNbZ+V4Dtqe4tmFAc4RC72JzJvhcRzufn5pkCGJZGCeCzXd0AvPqsJXV93JhWgMGEYhpTM1zKFb29RiNLCmA6FMCwjA7gs13dWFMTC85o8zRR+IIJRdZiej5xSFdGKzDfFMB0NEoAt2x+jDmLF9BzQth/GfS5AgwmFH0c9D1CcGKMHyiAUt7oAH6j63kWdyyOIIy9PFbx6znOTfWRgwnFVE5gFgfYEeFyeT6v8z1CUBTAsPgM4MDAALue7OG8y+J4uw5fsQsmFEMWR3gCbgeP+R4hEwpgdRTA9D1653oOv/wKL7fsS/1imEZaAQYXihjFGD+IM4CKX/V8BfD36zbxzT+7DYDCP9zDu//DlZzbeU6Kj5Dni2DekOregglFPwd5iUd8j3GcE6nuPWMaSZwBjPPpT4gvgr8tbKTv8OAnDPb39fPbwsaUQxHvKnC0YELRwjROocP3GMd5JofxSoMCOL44AxjfCvD0pBVrNlyfo2VqC+cny3yPFKxgQpFXeYxXGmIMoOJXXmwBXNy5mG8v+xnPP7ubv/r+hzm5s58XI4phPVeAwYRigF4O8lCmjzGDCzPdf0jiDGAvz9Dle4jUKYDj2/PcS7zhkmVc0XmJ71EyUO5p0MtSfaRgQtHC9DoUtDeqv3EMUQCHKYDhmGwAX+49xAtP7+H0jnjPI9RrFRhMKOolthN6AC9mvBLzRQEcpgAeb+uWZwGY13E4dxfChLYKDCYUjiL9FCa1j2aSVGYJTYzxAwWwEUwmgL/t2gvAirNWcAqnpTVSKrI/B/hHqe4tmFA0M4MTWTGpfbw0ydDklQIYFwUwHd1duzAzTnvNwro+biVCWwEGE4o0TDY0eRVjABs1fqAApmXr5sdYeHorA9MfyfSd4hphBRhMKIz9NA38W1X3GWi6PKNp8iXOABZ5iQ2+h8hEo0aw3gF8putOlna0Zf64+VwB/lmqewsmFE3MYjqdVd3n5SrDEoJGiR/EGkCtAOvBOceOLS/w+lVLM3+sWFeAIwUTilpUG5YgDPTyMut8T5E6BTB0+VoBPvvMQQ72vkJ7x35dBJOCcELRv58p+++t6i59r7o0o2H8UgDDoQD68buuLQCcd9ZKTmRyq4owV4AfTXVvwYTCaGWKXVzdnaoMSwhijR/EGcAYn/6E/Adwa9eLACztmPznN+QpgL4EE4paVB2WEEQYP4g3gDHGD8j9CnDrE5uYNauZRQv/HRuwqu6b9wj6EE4ojhTh+ULt91+QpDWJV1HGDxTAAOU5gtu7trD0zJOZYRdVfd8oVoFNn0p1d+GEoqkVpiW1338ykcmrSOIHCmBo8h7ALVv2cGFnW033zXMAfQknFJM1mcjkVYzxAwUwBDkO4Msv99Oz8yU+/P4TdQFMSsIJxeEiPF1If7+nJunvs15ijB8ogAHIcwB3bH8e59Zy9tIVTLHXVnfnHAewKifekOruvIbCzBYAv3TOXTDhNze3wuwk/SGyiI9vIccPFMDQ5CyAXVt3A3DW0nlV3zfPAfTJ94riq8B0rxNkER/fisC+gu8p0qcA5lPOAnjPzwffmnzv9nWwYE7tO8pZAH3yFgozuxToBZ6r6A6Hi7C9kOVI41uS+HncyVAAw6EApmbdhh7u+MH9ALzrY5tY838+SOeK2k5q5y2AVZmd7u7qEgozux2OeV/de4FLgPcCd5a53zXANQCnLZwN85MMpyzDV6CytiTxPUH1YgxgjE9/gpcAFh7qZmDAAXD4SD+Fh7prD0WOAuhbXULhnLt25G0z+zxwi3PuJbPxXwzjnFsNrAZYefYil+mQ5fgKVNYUwHyIMX7gJYDJGfuYdkITh48MMHWKkZyxN/05Ql8B1sCcq//PXzO7Hxgo3VwG/MA59xfl7rOyvdWtvyGgl9Kfm/ieoHG9WPA9QTaWJL4nCMK6jT0UHu4meX07nctqXE1MJO9PgV5UOPqlmW1wzq2czO68hOKYAcwKzrlkou9beeYit/7ma+owUUpifDpB8fNLAZRKnXvT0S/TCIXvq56oJBIAHCzCxsKx25ZVdlcvYlye/r7ge4JshBJAPQUaliWJ7wlS4z0UFTuhFc5Ijt02OhwxUPzqTwH0SwFM37np7i6cUIxldDhiUAS2FXxPkT4FsP4UQL8iCmA4oegtwm8K6e/3jUn6+5ysGAMY4+oPFEAfFMC6CycU01rhnCT9/RaBTYX09+tb3gIYY/xAAfQh1gDuI72LYC5MZzdDwglFlrIIkG9ZrL7yQAGsDwXQj5xGMJxQHCjCfQXfU0zs4sT3BINijB8ogPWiAIYl4wCGE4oZrXBB4nuKie0HHi34niJ9CmC2FMD6aJQAXpHu7oMJhdtf5NA9hYq+d9pbk0xnmVAIQatWCKu5WiiA2VIA6yPjAAYTCma2woVJRd9aaVBC4zWAMcYPFMCsKYB+/Em6uwsnFNWoMCihUQAzoACGRQH0IphQHCkW2X1vwfcYR827NKn/gyqAwdDTnxlRAL0IJhQ2q5Upb0l8j3FUnqKVFi/xgygDeKgIPFTwPUbqFMCMpB3AP093d8GE4nCxyM5Coez3nJYkdZkFyFW00vJSH/TdX/A9RuoUwPQogBnJeQCDCUVzayuzJgjBRCEJUT3jB3EGMMbVHyiAaYotgNP+W7r7CyYUlZgoJCHaCxxQACclxviBApi6CAOYlmBCcahY5Mkc/sBcWocfeDEGMMbVHyiAaVAA8yeYUExpbWV+Dn9g5jFeY6lH0KoRY/wgrADW+2nNSsUawHqeAzz1c+nuz/tHoVbKzIpAl+85cmIesNv3EDmhYzFMx2KYjsWwDudc62R2EMyKAuia7Oe+xsLM1utYDNKxGKZjMUzHYpiZrZ/sPprSGEREROKlUIiISFkhhWK17wFyRMdimI7FMB2LYToWwyZ9LII5mS0iIn6EtKIQEREPFAoRESkrd6Ews9lm9gsz+5WZ/cjMppa2f8PM1pnZ50Z873HbYmNmC8xs7YjbS8xsjZltNLMbSttazOynZvaAmV3tb9psjT4WI7b/1MyWlb5u9GPxWjO7p/R1Qx4LMzvNzApmdq+ZrbZBjXosjvtz13IschcK4APA151zbwOeA640s/cBzc65TmCJmS0da5vHmTNhZnOAfwFmjtj8X4DPO+eWAVeY2XzgvwIbnHNvAv7EzCb14po8GudYYGYfALY55zaWNjXysTDg60BLaVOjHotrgb90zl0KtAGvo3GPxVh/7qqPRe5C4Zy7xTl3T+nmfOAFIAG+V9r2K+DN42yLTT9wFbB/xLY9wHlmtgA4AXiJY4/F/UCMLzQ67liY2UnA14A/mNklpc0JDXgsSv4c+PWI2wkNeCycc9c7554o3ZzL4Cu0ExrwWDD2n3usbWV5f2W2md0OdIzYdK9z7otm1gnMcc49ZGYfBZ4p/fO9wHIGqzl6W9DKHIuR3/ZL4JPAqcC9QB/HH4sF2U+brQqPxV8D3wduB/5n6W9GDXkszGwu8J+AK0q/oEGPxYjvvQr4vXNul5k16rEY689d9bHwHgrn3LWjt5X+pngz8MelTQeA6aWvZzG4EhprW9DGOhZjuA74U+ecM7P/DbyV4WOxj8FjcSC7KeujwmNxAfBp59xzZvY9GvtYfAn4jHPuyIgfFI16LDCzJcCngctLmxr1WIz15676WOTuh2vp5PX3GfyP/qnS5g0MP7V0PtA9zrZGsBhoM7NpDK6iHI17LLYCS0pfrwSeonGPxcXAl82sACwzs7+nQY9F6bn6bwNXO+f2lTY35LEgpZ+d3lcUY/gIgz8Arzez64FbgTuBtWa2CHg7cCGDPyBHb2sENwIFBs/f/IzBp5+2AD83s1XAOcBvvE1XX18B/qn038lB4H3ASTTgsXDOnTn0tZkVnHOfM7PTacBjweCq+zTg5tLq6kYGT/I24rEY68/9zBjbygrmldmlvyW8FbjfOffceNsaVSmYbwbuHvG3qIakYzFMx2JYox6Lsf7c1R6LYEIhIiJ+5O4chYiI5ItCISIiZSkUIiJSlkIhIiJlKRQiIlKWQiFSIzM708ycmb3d9ywiWVIoRGo39P5iD3udQiRjCoVI7VYA3c653b4HEcmSQiFSuxXAw2Z2jZltMrPe0gdpBf9OxiIj6T6XDTkAAAEGSURBVJXZIjUysz8w+B5TjzD4nmQzgS8z+DkhZzrnej2OJ5KaPL4poEjumdkZwIkMvqHae1zpb1xmVgR+AbyJwQ/UEgmennoSqc2K0u/Xu2OX5Y+Vfp9f53lEMqNQiNRmOYOf1b1h1PZXl35/us7ziGRGoRCpzQpg1xjbr2Lwk8PW1XcckezoHIVIbZYDh8xsinOuD8DMTgU+BnzNOXfY63QiKdJVTyJVMrN2YEfp14PAPwOnAjcx+OlhlyoUEhM99SRSvaET2e9i8EPqfwp8CfghcIUiIbHRikJERMrSikJERMpSKEREpCyFQkREylIoRESkLIVCRETKUihERKQshUJERMr6/zf0NEiT+KyCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 这里给b和w不同的learning rate \n",
    "\n",
    "# y_data = b + w * x_data\n",
    "b = -120 # initial b\n",
    "w = -4 # initial w\n",
    "lr = 1 # learning rate 放大10倍\n",
    "iteration = 100000 # 这里直接规定了迭代次数，而不是一直运行到b_grad和w_grad都为0(事实证明这样做不太可行)\n",
    "\n",
    "# store initial values for plotting，我们想要最终把数据描绘在图上，因此存储过程数据\n",
    "b_history = [b]\n",
    "w_history = [w]\n",
    "\n",
    "lr_b = 0\n",
    "lr_w = 0\n",
    "\n",
    "# iterations\n",
    "for i in range(iteration):\n",
    "    \n",
    "    # get new b_grad and w_grad\n",
    "    b_grad,w_grad=getGrad(b,w)\n",
    "    \n",
    "    # get the different learning rate for b and w\n",
    "    lr_b = lr_b + b_grad ** 2\n",
    "    lr_w = lr_w + w_grad ** 2\n",
    "    \n",
    "    # 这一招叫做adagrad，之后会详加解释\n",
    "    # update b and w with new learning rate\n",
    "    b -= lr / np.sqrt(lr_b) * b_grad\n",
    "    w -= lr / np.sqrt(lr_w) * w_grad\n",
    "    \n",
    "    #store parameters for plotting\n",
    "    b_history.append(b)\n",
    "    w_history.append(w)\n",
    "    \n",
    "    # output the b w b_grad w_grad\n",
    "    # print(\"b: \"+str(b)+\"\\t\\t\\t w: \"+str(w)+\"\\n\"+\"b_grad: \"+str(b_grad)+\"\\t\\t w_grad: \"+str(w_grad)+\"\\n\")\n",
    "    \n",
    "# output the final function and its error\n",
    "print(\"the function will be y_data=\"+str(b)+\"+\"+str(w)+\"*x_data\")\n",
    "error=0.0\n",
    "for i in range(10):\n",
    "    print(\"error \"+str(i)+\" is: \"+str(np.abs(y_data[i]-(b+w*x_data[i])))+\" \")\n",
    "    error+=np.abs(y_data[i]-(b+w*x_data[i]))\n",
    "average_error=error/10\n",
    "print(\"the average error is \"+str(average_error))\n",
    "\n",
    "# plot the figure\n",
    "plt.contourf(x,y,Z,50,alpha=0.5,cmap=plt.get_cmap('jet'))\n",
    "plt.plot([-188.4],[2.67],'x',ms=12,markeredgewidth=3,color='orange')\n",
    "plt.plot(b_history,w_history,'o-',ms=3,lw=1.5,color='black')\n",
    "plt.xlim(-200,-100)\n",
    "plt.ylim(-5,5)\n",
    "plt.xlabel(r'$b$',fontsize=16)\n",
    "plt.ylabel(r'$w$',fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 有了新的learning rate以后，从初始值到终点，我们在100000次iteration之内就可以顺利地完成了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
